Twitter announced this week that they were launching there own URL shortener.   There has been a lot of chatter about this over the past week.  I thought it would be helpful to write a about how the partnership worked and what bit.ly relationship is to platforms, Twitter and others.    To do something unusual for me let me let me cut to the chase.

Twitter.com pretty much stopped using bit.ly to shorten URL’s on Twitter.com in December.    Since last fall the bit.ly team and Twitter have been talking about this transition.    Today Twitter.com represents less than 1% of bit.ly links shortened — when the transition took place in December it was closer to 3-8%, depending on the UX on Twitter.com and the day.   We continue to work with the Twitter team and we are currently figuring out how to get key whitelabel URL’s working on Twitter.com.    The default shortening partnership worked well for a period of time – approximately six months — during a period of hyper growth. Today bit.ly is growing and continues to scale — irrespective of the change in rules last December re: shortening on Twitter.com.   That is the summary — the detailed version follows.

bit.ly was launched May of 2008.    By the first quarter of 2009 bit.ly was growing fast, scaling well and offering a handful of key features beyond shortening that users – of both the api and the website – found critical in terms of understanding social distribution — most importantly real time metrics*.    I believe Twitter’s insane growth trajectory started in December 2008 — by early 2009 many of the short URL’s on Twitter were struggling to keep up with the scale and growth and none of them offered the real time metrics that bit.ly had.   So Twitter and bit.ly entered into an agreement where bit.ly would become the default URL shortener for Twitter.     This feature rolled out in May 2009 and ran until December of 2009.

bit.ly knew this would be a short term agreement — it was done to help Twitter scale and without a doubt it helped bit.ly scale.     In late November / December 2009 Twitter.com stopped shortening URL’s — except under one very narrow use case (and if you can find out what that is I will send or buy you a drink!).    As Techcrunch reported this week bit.ly growth has continued.

When Twitter changed its shortener policy in December Twitter.com represented 3-8% of bit.ly links created everyday.    So the change was barely noticeable in bit.ly systems.    Today Twitter.com represents less than .5% of bit.ly links created or clicked on each day.     There are other social platforms that are now larger than Twitter.com.    Last month there were  3.4bn clicks on bit.ly links — up from 2.7bn in February and 2.5bn in January.    bit.ly is fairly big for a little company, handling billions of clicks and real time metrics for 100′s of million URL’s each day isnt trivial.    Someone noted earlier this week — they believe — Yahoo does about 7.5bn clicks a month on its search product — while these clicks are not comparable to the bit.ly click experience, in terms of reach and scale it’s an interesting benchmark.

On Tuesday we announced 6,000 sign up’s for bit.ly pro.   As of today that number is over 7,000 and have in the past 48hrs a subsset have signed up for the enterprise version — so revenue.    The companies up and running include: nyti.ms, amzn.to, binged.it, huff.to, 4sq.com, pep.si, and n.pr — along side a set of bloggers and individuals who use the bit.ly service for their URL’s.    And incidentally — this Wednesday was our first day ever where over 150m bit.ly links were clicked on.          (For more data and charts of historical growth see )

All that said the noise level out there is well, noisy — “is bit.ly screwed?”, “is bit.ly the next google?” seemingly, no one can make up their mind.    We can — we love bit.ly.    bit.ly is short, sweet and out of control.   Someone asked me last summer, “is bit.ly part of the internet”.     We are working hard to make it part of the internet or at least the social, real time web — in scale, breadth, trust and performance.  bit.ly is the tracking tool that many many people use to understand how many times a Tweet or a Facebook link was clicked on*.

We thank Twitter, everyone there, for the kick start it gave bit.ly.   And we certainly hope we helped Twitter during a difficult scaling period — that was the intent.    bit.ly still works and will continue to work on Twitter, most of the clients and Twitter related services use the API everyday and we are working right now with the Twitter team on some publisher related services.   And most of all we thank our users — end users who use the bit.ly web site to shorten, share and track everyday — bit.ly 1.3 will be out in the next few days and we hope you love it.   And we thank our API users.    The myriad of services who use our API to shorten and track and monitor the pulse of the real time web.   And publishers who are using it for domain level / enterprise tracking.

In terms of lessons learnt there are many — but four come to mind right now, all four relate to broader points about web platforms.   Over the past few years a set of platforms have emerged online that give start-up’s a foundation to get a kick start to building their audience and/or their business.   Adwords/Adsense were probably the first scaled examples of this. And as these platforms mature its important for their to be clear boundaries between what the platform provider does and doesn’t do. Granted these boundaries shift over time — but the boundaries have to be sustained for long enough for the platform provider to achieve scale and trust and to get a critical mass of applications running on it.  They also have to sustained long enough for businesses to be built on the platform, not just tweaks, real businesses.

To play out the Google example take the UX of Google.   Google understood they werent in the content business — they were in the navigation business.   So for years the Google site just pointed outward.   Now after 10 years the line is getting hazy in some areas — this is why the local search stuff, the yelp conversations resonate with people — Google has for what ever reason decided that local is something it needs to wrap more of an arm around local. How long is that arm? How detrimental is it to local players? i’m not sure? — but if i had to put a dollar down I would bet that Yelp and say Opentable will do just fine.    So — clear sustained boundaries are necessary.    The second point is that these boundaries become increasingly important and easy to define once the monetization approach of the underlying platform is defined.   Emphasis is the reason why this is seperate point to the first one — vs. a subset, this is vital.   The third point is that people bootstrapping on these platforms should also try to spread their relevance beyond a single platform – so Yelp should extend its business model beyond adsense, Zynga beyond Facebook etc. etc.    In 2010, unlike 10 years ago, we are building in a world of multiple, often overlapping platforms, its not a monolithic world anymore.    That is what Stocktwits has done, same for bit.ly, Tweetdeck, Someecards, OMGpop etc… all of these services have a leg in multiple platforms.

Lastly, talk about holes and filling holes in platforms is misleading at best.    Take a list of emerging to mature companies — great companies … Is Groupon a hole in Facebook? Facebook a hole in Google?? Google is a hole in Microsoft???  Microsoft in IBM????  Maybe it’s holes all the way down?    Innovation — building great companies — is about finding, filling and even creating holes.   But entrepreneurs should n’t — and most dont — focus on filling holes in other people’s platforms — they should think about how to build great things — things that in 2010 may be bootstrapped on platforms but great products, products that people love, products that move people to organize their world differently, or to see the world differently.   The slogan “Think different” captured most if not all of what entrepreneurs need.   After 30yrs of personal computing history we have a lot of platform and application history to draw from — Apple understands this very well, so does Google,  same for Microsoft, Amazon, and Ebay.  And yes — once again, the cycle of innovation is turning – great new platforms are emerging and great businesses will be developed on of these new platforms.

*/ note:  if you place a “+” on the end of any bit.ly link and you will see real time traffic to that link

Tracking the real time web data

The measurement tools we have still only sometimes work for counting traffic to web pages and they certainly dont track or measure traffic in streams let alone aggregate up the underlying ecosystems that are emerging around these new markets.  At betaworks we spend a lot of time looking at and tracking this underlying data set.   It’s our business and its fascinating.   Like many companies each of the individual businesses at betaworks have fragments of data sets but because betaworks acts as ecosystem of companies we can mix and match the data to get results that are more interesting and hopefully offer greater insight

——————————-

(i) tumblr growth for the last half of 2009

Another data point re: growth of the real time web through the second half of last year through to Jan 18th of this year.  tumblr continues to kill it.     I read this interesting post yesterday about how tumblr is leading in its  category through innovation and simple, effective, product design.   The compete numbers quoted in that post are less impressive than these directly measured quantcast numbers.

(h) Twitter vs. the Twitter Ecosystem

Fred Wilson’s post adds some solid directional data on the question of the size of the ecosystem.   “You can talk about Twitter.com and then you can talk about the Twitter ecosystem. One is a web site. The other is a fundamental part of the Internet infrastructure. And the latter is 3-5x bigger than the former and that delta is likely to grow even larger.”

(g) Some early 2010 data points re: the Real Time Web

• Twitter: Jan 11th was the highest usage day ever (source: @ev via techcrunch)
• Tweetdeck: did 4,143,687 updates on Jan 8, yep 4m. Or, 48 per second (source: Iain Dodsworth / tweetdeck internal data)
• Foursquare: Jan 9th biggest day ever.    1 update or check-in per second (source: twitter and techcrunch)
• Daily Booth: in past 30 days more than 10mm uniques (source: dailybooth internal data)
• bit.ly: last week was the largest week ever for clicks on bit.ly links. 564m were clicked on in total. On the Jan 6th there were a record of 98m decodes.    1100 clicks every second.

(f) Comparing the real time web vs. Google for the second half of 2009

Andrew Parker commented on the last post that the chart displaying the growth trends was hard to decipher and that it maybe simpler to show month over month trending.  It turns out the that month over month is also hard to decipher.   What is easier to read is this summary chart.    It shows the average month over month growth rates for the RT web sites (the average from Chart A).   Note 27.33% is the average growth rate for the real time web companies in 2009 — that’s astounding.    The comparable number for the second half of 2009 was 10.5% a month — significantly lower but still a very big number for m/m growth.

(e) Ongoing growth of the real time stream in the second half of 2009

This is a question people have asked me repeatedly in the past few weeks.  Did the real time stream grow in Q4 2009?    It did.    Not at the pace that it grew during q1-q3, but our data at betaworks confirms continued growth.   One of the best proxies we use for directional trending in the real time web are the bit.ly decodes.   This is the raw number of bit.ly links that are clicked on across the web.    Many of these clicks occur within the Twitter ecosystem, but a large number are outside of Twitter, by people and by machines — there is a surprising amount of diversity within the real time stream as I posted about a while back.

Two charts are displayed below.    On the bottom are bit.ly decodes (blue) and encodes (red)  running through the second half of last year.    On the top is a different but related metric.   Another betaworks company is Twitterfeed.    Twitterfeed is the leading platform enabling publishers to post from their sites into Twitter and Facebook.    This chart graphs the total number of feeds processed (blue) and the total number of publishers using Twitterfeed, again through the second half of the year (note if the charts inline are too small to read you can click though and see full size versions).   As you can see similar the left hand chart — at Twitterfeed the growth was strong for the entire second half of 2009.

Both these charts illustrate the ongoing shift that is taking place in terms of how people use the real time web for navigation, search and discovery.    My preference is to look at real user interactions as strong indicators of user behavior.   For example I actually find Google trends more useful often than comScore, Compete or the other “page” based measurement services.   As interactions online shift to streams we are going to have to figure out how measurement works. I feel like today we are back to the early days of the web when people talked about “hits” — it’s hard to parse the relevant data from the noise.  The indicators we see suggest that the speed at which this shift to the real time web is taking place is astounding.   Yet it is happening in a fashion that I have seen a couple of times before.

(d) An illustration of the step nature of social growth. bit.ly weekly decodes for the second half of 2009.

Most social networks I have worked with have grown in a step function manner.  You see this clearly when you zoom into the bit.ly data set and look at weekly decodes, illustrated above.   You often have to zoom in and out of the data set to see and find the steps but they are usually there.     Sometimes they run for months — either up or sideways.    You can see the steps in Facebook growth in 2009.    I saw effect up close with ICQ, AIM, Fotolog, Summize and now with bit.ly.   Someone smarter than me has surely figured out why these steps occur.    My hypothesis is that as social networks grow they jump in a sporadic fashion from one dense cluster of relationships to a new one.   The upward trajectory is the adoption cycle of that new, dense cluster and the flat part of the step is the period between the step to next cluster.     Blended in here there are clearly issues of engagement vs. trial.   But it’s hard to weed those out from this data set.   As someone mentioned to me in regards to the last post this is a property of scale-free networks.

(c) Google and Amazon in 2009

Google and Amazon — this is what it looked like in 2009:

It’s basically flat.     Pretty much every user in the domestic US is on Google for search and navigation and on Amazon for commerce — impressive baseline numbers but flat for the year (source: Quantcast).  So then lets turn to Twitter.

(b) Twitter – an estimate of Twitter.com and the Twitter ecosystem

Much ink has been spilt over Twitter.com’s growth in the second half of the year.   During the first half of the year Twitter’s experience hyper growth — and unprecedented media attention.    In the second half of the year the media waned, the service went through what I suspect was a digestion phase — that step again?     Steps aside — because I dont in anyway seek to represent Twitter Inc. — there are two questions that in my mind haven’t been answered fully:

(i) what international growth in the second half of 2009?, that was clearly a driver for Facebook in ’09.  Recent data suggests growth continued to be strong.

(ii) what about the ecosystem.

Unsurprisingly its the second question that interests me the most.    So what about that ecosystem?    We know that approx 50% of the interactions with the Twitter API occur outside of Twitter.com but many of those aren’t end user interactions.     We also know that as people adopt and build a following on Twitter they often move up to use one of the client or vertical specifics applications to suit their “power” needs.   At TweetDeck we did a survey of our users this past summer.     The data we got suggested 92% of them then use Tweetdeck everyday — 51% use Twitter more frequently since they started using TweetDeck.  So we know there is a very engaged audience on the clients.     We also know that most of the clients arent web pages — they are flash, AIR, coco, iPhone app’s etc. all things that the traditional measurement companies dont track.

What I did to estimate the relative growth of the Twitter ecosystem is the following.   I used Google Trends and compiled data for Twitter and the key clients.    I then scaled that chart over the Twitter.com traffic.   Is it correct? — no.   Is it made up? — no.   It’s a proxy and this is what it looks like (again, you can click the chart to see a larger version).

Similar to the Twitter.com traffic you see the flattening out of the ecosystem in the summer.    But you see growth in the forth quarter that returns to the summer time levels.     I suspect if you could zoom in and out of this the way I did above you would see those steps again.

(a) The Real Time Web in 2009

Add in Facebook (blue) and Meebo (green) both steaming ahead — Meebo had a very strong end of year.    And then tile on top the bit.ly data and the Twitterfeed numbers (bit.ly on the right hand scale) and you have an overall picture of growth of the real time web vs. Google and Amazon.   As t

For a while now I have been thinking about doing a post about some of the data we track at betaworks.   Over the past few months people have written about Twitter’s traffic being up, down or sideways — the core question that people are asking is the real time web growing or not, is this hype or substance?     Great questions — the answer to all of the above is from the data set I see: yes.   Adoption and growth is happening pretty much across the board — and in some areas its happening at an astounding pace.    But tracking this is hard.   It’s hard to measure something that is still emerging.    The measurement tools we have still only sometimes work for counting traffic to web pages and they certainly dont track or measure traffic in streams let alone aggregate up the underlying ecosystems that are emerging around these new markets.  At betaworks we spend a lot of time looking at and tracking this underlying data set.   It’s our business and its fascinating.

I was inspired to finally write something by first a good experience and then a bad one.    First the good one.    Earlier this week I saw a Tweet from Marshall Kirkpatrick about Gary Hayes’s social media counter.    It’s  very nicely done — and an embed is available.     This is what it looks like (note the three buttons on top are hot, you can see the social web, mobile and gaming):

The second thing was less fun but i’m sure it has happened to many an entrepreneur.    I was emailed earlier this week by a reporter asking about some data – I didnt spend the time to weed through the analysis and the reporter published data that was misleading.    More on this incident later.

Lets dig into some data.    First — addressing the question people have asked me repeatedly in the past few weeks.  Did the real time stream grow in Q4 2009?    It did.    Not at the pace that it grew during q1-q3, but our data confirms continued growth.   One of the best proxies we use for directional trending in the real time web are the bit.ly decodes.   This is the raw number of bit.ly links that are clicked on across the web.    Many of these clicks occur within the Twitter ecosystem, but a large number are outside of Twitter, by people and by machines — there is a surprising amount of diversity within the real time stream as I posted about a while back.  Two charts are displayed below.    On the left there are bit.ly decodes (blue) and encodes (red)  running through the second half of last year.    On the right is a different but related metric.   Another betaworks company is Twitterfeed.    Twitterfeed is the leading platform enabling publishers to post from their sites into Twitter and Facebook.    This chart graphs the total number of feeds processed (blue) and the total number of publishers using Twitterfeed, again through the second half of the year (note if the charts inline are too small to read you can click though and see full size versions).   As you can see similar the left hand chart — at Twitterfeed the growth was strong for the entire second half of 2009.

Both these charts illustrate the ongoing shift that is taking place in terms of how people use the real time web for navigation, search and discovery.    My preference is to look at real user interactions as strong indicators of user behavior.   For example I actually find Google trends more useful often than comScore, Compete or the other “page” based measurement services.   As interactions online shift to streams we are going to have to figure out how measurement works. I feel like today we are back to the early days of the web when people talked about “hits” — it’s hard to parse the relevant data from the noise.  The indicators we see suggest that the speed at which this shift to the real time web is taking place is astounding.   Yet it is happening in a fashion that I have seen a couple of times before.

Most social networks I have worked with have grown in a step function manner.  You see this clearly when you zoom into the bit.ly data set and look at weekly decodes.   This is less clear but also visible when you look at daily trending data (on the right) — but add a 3 week moving average on top of that and you can once again see the steps.   You often have to zoom in and out of the data set to see and find the steps but they are usually there.     Sometimes they run for months — either up or sideways.      I saw this with ICQ, AIM, Fotolog, Summize through to bit.ly.   Someone smarter than me has surely figured out why these steps occur.    My hypothesis is that as social networks grow they jump in a sporadic fashion to a new dense cluster or network of relationships.   The upward trajectory is the adoption cycle of that new, dense cluster and the flat part of the step is the period between the step to next cluster.     Blended in here there are clearly issues of engagement vs. trial.   But it’s hard to weed those out from this data set.   I learnt a lot of this from Yossi Vardi and Adam Seifer.    Two people I had the privilege of working with over the years — two people whose DNA is wired right into this stuff.  At Fotolog Adam could take the historical data set and illustrate how these clusters moved — in steps — from geography to geography, its fascinating.

TechCrunch falls off a cliff

Ok I’m sure there are some people reading who are thinking — well this is interesting but I actually want to read about TechCrunch falling off a traffic cliff.   I’m sorry – I actually don’t have any data to suggest that happened.  After noting yesterday that provocative headline is  sometimes a substitute for data I thought — heck I can do this too!    This section of the post is more of a cautionary tale — if you are confused by this twist let me back up to where I started.   I mentioned that there were two motivations for me sitting down and writing this post.   The second one was that earlier this week  TechCrunch story ran this week saying that bit.ly market share had shifted dramatically.     It hasn’t.   The data was just misunderstood by the reporter.   The tale (I did promise a tale) began last August when TechCrunch ran the following chart about the market share of URL shorteners.

The pie chart showed the top 5 URL shorteners and then calculated the market share each had  — what percent each was *of* the top five.     The  data looks like this:

bit.ly 79.61%
TinyURL 13.75%
is.gd 2.47%
ow.ly 2.26%
ff.im 1.92%
(79.61+13.75+2.47+2.26+1.92 = 100)

The comparable data from yesterday is:

bit.ly = 75%
TinyURL = 10%
ow.ly = 6%
is.gd = 4%
tumblr = 4%
(again this adds up to 100%)

Not much news in those numbers, especially when you consider they come from the Twitter “garden hose” (a subset of all tweets) and swing by as much as +/- 5% daily.   The tumblr growth into the top 5 and the ow.ly bump up is nice shift for them – but not really a story.     The hitch was that the reporter didn’t consider that there are other URL’s in the Twitter stream aside from these five.   Some are short URL’s and some aren’t.   So this metric doesn’t accurately reflect overall short URL market share — it shows the shuffling of market share amongst the top five.   But media will be media.   I saw a Tweet this week about how effective Twitter is at disseminating information — true and false — despite all the shifts that are going on headlines in a sense carry even more weight than in the “read all about it” days.

The lesson here for me was the importance of helping reporters and analysts get access to the underlying data — data they can use effectively.   We sent the reporter the  data but he saw a summary data set that included the other URL’s and didn’t understand that back in August there were also “other” URL’s.   After the fact we worked to sort this out and he put a correction in his post.   But the headline was off and running — irrespective of how dirty or clean the data was.   Basic mistake — my mistake — and this was with a reporter who knows this stuff well.   Given the paucity of data out there and the emergent state of the real time web  this stuff is bound to happen.

Ironically, yesterday, bit.ly hit an all time high in terms of decodes — over 90m.   But back to the original question — there is a valid question the reporter was seeking to understand, namely: what is the market share of dem short thingy’s?      We track this metric — using the Twitter garden hose and identifying most of the short URL’s to produce a ranking (note its a sample, so the occurrences are a fraction of the actuals).     And it’s a rolling 24 hr view — so it moves around quite a bit — but nonetheless it’s informative.  This is what it looked like yesterday:

Over time this data set is going to become harder to use for this purpose.    At bit.ly we kicked off our white label service before the holidays.   Despite months of preparation we weren’t expecting the demand.   As we provision and setup the thousands of publishers, blogger and brands who want white label services its going to result in a much more diverse stream of data in the garden hose.

Real Time Web Data

Finally I thought it would be interesting to try to get a perspective on the emergence of the real time web in 2009 — how did its growth compare and contrast with the incumbent web category leaders?    Let me try to frame up some data around this.   Hang in there, some of the things I’m going to do are hacks (at best) — as I said I was inspired!   Lets start with the user growth in the US among the current web leaders — Google and Amazon — this is what it looked like in 2009:

It’s basically flat.     Pretty much every user in the domestic US is on Google for search and navigation and on Amazon for commerce — impressive baseline numbers but flat for the year (source: Quantcast).  So then lets turn to Twitter.    Much ink has been spilt over Twitter.com’s growth in the second half of the year.   During the first half of the year Twitter’s growth, I suspect, was driven to a great extent by the unprecedented media attention it received — media and celebrities were all over it.    Yet in the second half of the year that waned and the traffic numbers to the Twitter.com web site were flat for the second half of the year.    That step issue again?

Placing steps aside — because I dont in anyway seek to represent Twitter Inc. — there are two questions that haven’t been answered  (a) what about international growth, that was clearly a driver for Facebook in ’09, where was Twitter internationally?   (b) what about the ecosystem.     Unsurprisingly its the second question that interests me the most.    So what about that ecosystem?

We know that approx 50% of the interactions with the Twitter API occur outside of Twitter.com but many of those aren’t end user interactions.     We also know that as people adopt and build a following on Twitter they often move up to use one of the client or vertical specifics applications to suit their “power” needs.   At TweetDeck we did a survey of our users this past summer.     The data we got suggested 92% of them then use Tweetdeck everyday — 51% use Twitter more frequently since they started using TweetDeck.  So we know there is a very engaged audience on the clients.     We also know that most of the clients arent web pages — they are flash, AIR, coco, iPhone app’s etc. all things that the traditional measurement companies dont track.

What I did to estimate the relative growth of the Twitter ecosystem is the following.   I used Google Trends and compiled data for Twitter and the key clients.    I then scaled that chart over the Twitter.com traffic.   Is it correct? — no.   Is it made up? — no.   It’s a proxy and this is what it looks like (again, you can click the chart to see a larger version):

Similar to the Twitter.com traffic you see the flattening out in the summer.    But similar to the data sets referenced above you see growth in the forth quarter.     I suspect if you could zoom in and out of this the way I did above you would see those steps again.     So lets put it all together!    Its one heck of a busy chart.   Add in Facebook (blue) and Meebo (green) both steaming ahead — Meebo had a very strong end of year.    And then tile on top the bit.ly data and the Twitterfeed numbers (both on different scales) and you have an overall picture of growth of the real time web vs. Google and Amazon.

Ok.   One last snap shot then im wrapping up.    Chartbeat — yep another betaworks company — had one of its best weeks ever this past week — no small thanks to Jason’s Calacanis’s New Year post about his Top 10 favorite web products of 2009.   To finish up here is a video of the live traffic flow coming into Fred Wilson’s blog at AVC.com on the announcement of the Google Nexus one Phone.    Steve Gilmore mentioned the other week how sometimes interactions in the real time web just amaze one.    Watching people swarm to a site is a pretty enthralling experience.    We have much work to do in 2010.    Some of it will be about figuring out how to measure the real time web.   Much of it will be continuing to build out the real time web and learning about this fascinating shift taking place right under our feet.

random footnote:

A data point I was sent this am by Iain that was interesting — yet it didnt seem to fit in anywhere?!   Asian twitter clients were yesterday over 5% of the requests visible in the garden hose.

As the title suggests the focus was on the Twitter ecosystem in London.    Our conversation also touched on the overall size and health of the real-time ecosystem — this topic didn’t make it into the article. It’s hard to gauge the health of a business ecosystem that is still very much under development and has yet to mature into one that produces meaningful revenues. Yet the question got me thinking — it also got me thinking that it has been a while since I had posted here. It was one busy summer. I have a couple of long posts I’m working on but for now I want to do this quick post on the real-time ecosystem and in it offer up some metrics on its health.

Back in June I did a presentation at Jeff Pulver’s 140conf, the topic of which was the real-time / Twitter ecosystem.   Since then, I have been thinking about the diversity of data sources, notably the question of where people are publishing and consuming real-time data streams. At betaworks we are fairly deep into the real time / Twitter ecosystem.  In fact, every company at betaworks is a participant, in one manner or another, in this ecosystem, and that’s a feature, not a bug! Of the 20 or so companies in the betaworks network, there is a subset that we we operate; one of those is bit.ly.

In an attempt to answer this question about the diversity of the ecosystem, let me run through some internal data from bit.ly.   bit.ly is a URL shortener that offers among other things real-time tracking of the clicks on each link (add “+” to any bit.ly URL to see this data stream).   With a billion bit.ly links clicked on in August — 300m last week — bit.ly has become almost part of the infrastructure of the real time cloud.  Given its scale bit.ly’s data is a fair proxy for the activity of the real-time stream, at least of the links in the stream.

On Friday of this week (yesterday) there were 20,924,833 bit.ly links created across the web (we call these “encodes”). These 20.9m encodes are not unique URL’s, since one popular URL might have been shortened by multiple people. But each encode represents intentionality of some form. bit.ly in turn retains a parent : child mapping, so that you can see what your sharing of a link generates vs. the population (e.g., I shared a video on Twitter the other day; my specific bit.ly link got 88 clicks, out of a total of 250 clicks on any bit.ly link to that same video.  see http://bit.ly/Rmi25+).

So where were these 20.9m encodes created? Approximately half of the encodes took place within the Twitter ecosystem. No surprise here: Twitter is clearly the leading public, real-time stream and about 20% of the updates on Twitter contain at least one link, approx half of which are bit.ly links.   But here is something surprising: less than 5% of the 20.9m came from Twitter.com (i.e., from Twitter’s use of bit.ly as the default URL-shortener). Over 45% of the total encodes came from other services associated in some way with Twitter – i.e. the Twitter ecosystem — a long and diverse list of services and companies within the ecosystem who use bit.ly.

The balance of the encodes came from other areas of the real time web, outside of Twitter. Google Reader incorporated bit.ly this summer, as did Nokia, CBS, Dropbox, and some tools within Facebook. And then of course people use the bit.ly web site — which has healthy growth — to create links and then share them via instant-messaging services, MySpace, email, and countless other communications tools.

The bit.ly links that are created are also very diverse. Its harder to summarise this without offering a list of 100,000 of URL’s — but suffice it to say that there are a lot of pages from the major web publishers, lots of YouTube links, lots of Amazon and eBay product pages, and lots of maps. And then there is a long, long tail of other URL’s. When a pile-up happens in the social web it is invariably triggered by link-sharing, and so bit.ly usually sees it in the seconds before it happens.

This data says to me that the ecosystem as a whole is becoming fairly diverse. Lots of end points are publishing (i.e. creating encodes) and then many end points are offering ways to use the data streams.

In turn, this diversity of the emerging ecosystem is, I believe, an indicator of its health. Monocultures aren’t very resilient to change; ecosystems tend to be more resilient and adaptable. For me, these few data points suggest that the real-time stream is becoming more and more interesting and more and more diverse.

Gottwald was flanked by his comrades, with Clementis standing next to him. There were snow flurries, it was cold, and Gottwald was bareheaded. The solicitous Clementis took off his own fur cap and set it on Gottwald’s head.

The Party propaganda section put out hundreds of thousands of copies of a photograph of that balcony with Gottwald, a fur cap on his head and comrades at his side, speaking to the nation. On that balcony the history of Communist Czechoslovakia was born. Every child knew the photograph from posters, schoolbooks, and museums.

Four years later Clementis was charged with treason and hanged. The propaganda section immediately airbrushed him out of history, and obviously, out of all the photographs as well. Ever since, Gottwald has stood on that balcony alone. Where Clementis once stood, there is only bare palace wall. All that remains of Clementis is the cap on Gottwald’s head.

Book of Laughter and Forgetting, Milan Kundera

The rise of social distribution networks

Over the past year there has been a rapid shift in social distribution online.    I believe this evolution represents an important change in how people find and use things online. At betaworks I am seeing some of our companies get 15-20% of daily traffic via social distribution — and the percentage is growing.    This post outlines some of the aspects of this shift that I think are most interesting.   The post itself is somewhat of a collage of media and thinking.

Distribution is one of the oldest parts of the media business.    Content is assumed to be king so long as you control the distribution flow to that content. From newspapers to NewsCorp companies have understand this model well.   Yet this model has never suited the Internet very well.     From the closed network ISP’s to Netcenter.   Pathfinder to Active desktop, Excite Lycos, Pointcast to the Network computer.   From attempts to differentially price bits to preset bookmarks on your browser — these are all attempts at gate keeping attention and navigation online.    Yet the relative flatness of the internet and its hyperlinked structure has offered people the ability to route around these toll gates.   Rather than client software or access the nexus of distribution became search.    Today there seems to be a new distribution model that is emerging.   One that is based on people’s ability to publically syndicate and distribute messages — aka content — in an open manner.    This has been a part of the internet since day one — yet now its emerging in a different form — its not pages, its streams, its social and so its syndication.    The tools serve to produce, consume, amplify and filter the stream.     In the spirit of this new wave of Now Media here is a collage of data about this shift.

Dimensions of the now web and how is it different?

Start with this constant, real time, flowing stream of data getting published, republished, annotated and co-opt’d across a myriad of sites and tools.    The social component is complex — consider where its happening.    The facile view is to say its Twitter, Facebook, Tumblr or FriendFeed — pick your favorite service.    But its much more than that because all these sites are, to varying degrees, becoming open and distributed. Its blogs, media storage sites (ie: twitpic) comment boards or moderation tools (ie: disqus) — a whole site can emerge around an issue — become relevant for week and then resubmerge into the morass of the data stream, even publishers are jumping in, only this week the Times pushed out the Times Wire.    The now web — or real time web — is still very much under construction but we are back in the dark room trying to understand the dimensions and contours of something new, or even to how to map and outline its borders. Its exciting stuff.

Think streams …

First and foremost what emerges out of this is a new metaphor — think streams vs. pages.     This seems like an abstract difference but I think its very important.    Metaphors help us shape and structure our perspective, they serve as a foundation for how we map and what patterns we observe in the world.     In the initial design of the web reading and writing (editing) were given equal consideration – yet for fifteen years the primary metaphor of the web has been pages and reading.     The metaphors we used to circumscribe this possibility set were mostly drawn from books and architecture (pages, browser, sites etc.).    Most of these metaphors were static and one way.     The steam metaphor is fundamentally different.  Its dynamic, it doesnt live very well within a page and still very much evolving.    Figuring out where the stream metaphor came from is hard — my sense is that it emerged out of RSS.    RSS introduced us to the concept of the web data as a stream — RSS itself became part of the delivery infrastructure but the metaphor it introduced us to is becoming an important part of our eveyday day lives.

A stream.   A real time, flowing, dynamic stream of  information — that we as users and participants can dip in and out of and whether we participate in them or simply observe we are are a part of this flow.     Stowe Boyd talks about this as the web as flow: “the first glimmers of a web that isnt about pages and browsers” (see this video interview,  view section 6 –> 7.50 mins in).       This world of flow, of streams, contains a very different possibility set to the world of pages.   Among other things it changes how we perceive needs.  Overload isnt a problem anymore since we have no choice but to acknowledge that we cant wade through all this information.   This isnt an inbox we have to empty,  or a page we have to get to the bottom of — its a flow of data that we can dip into at will but we cant attempt to gain an all encompassing view of it.     Dave Winer put it this way in a conversation over lunch about a year ago.    He said “think about Twitter as a rope of information — at the outset you assume you can hold on to the rope.  That you can read all the posts, handle all the replies and use Twitter as a communications tool, similar to IM — then at some point, as the number of people you follow and follow you rises — your hands begin to burn. You realize you cant hold the rope you need to just let go and observe the rope”.      Over at Facebook Zuckerberg started by framing the flow of user data as a news feed — a direct reference to RSS — but more recently he shifted to talk about it as a stream: “… a continuous stream of information that delivers a deeper understanding for everyone participating in it. As this happens, people will no longer come to Facebook to consume a particular piece or type of content, but to consume and participate in the stream itself.”    I have to finish up this section on the stream metaphor with a quote from Steve Gillmor.    He is talking about a new version of Friendfeed, but more generally he is talking about real time streams.     The content and the language — this stuff is stirring souls.

We’re seeing a new Beatles emerging in this new morning of creativity, a series of devices and software constructs that empower us with both the personal meaning of our lives and the intuitive combinations of serendipity and found material and the sturdiness that only rigorous practice brings. The ideas and sculpture, the rendering of this supple brine, we’ll stand in awe of it as it is polished to a sparkling sheen. (full article here)

Now, Now, Now

The real time aspect of these streams is essential.  At betaworks we are big believers in real time as a disruptive force — it’s an important aspect of many of our companies — it’s why we invested a lot of money into making bit.ly real time.  I remember when Jack Dorsey first saw bit.ly’s  plus or info page (the page you get to by putting a “+” at the end of any bit.ly URL) –  he said this is “great but it updates on 30 min cycles, you need to make it real time”.   This was August of ’08 — I registered the thought, but also thought he was nuts.    Here we sit in the spring of ’09 and we invested months in making bit.ly real time –  it works, and it matters.   Jack was right — its what people want to see the effects on how a meme is are spreading — real time.   It makes sense — watching a 30 min delay on a stream — is somewhere between weird and useless.   You can see an example of the real time bit.ly traffic flow to an URL  here. Another betaworks company, Someecards, is getting 20% of daily traffic from Twitter.   One of the founders Brook Lundy said the following “real time is now vital to what do.    Take the swine flu — within minutes of the news that a pandemic level 5 had been declared — we had an ecard out on Twitter”.    Sardonic, ironic, edgy ecards — who would have thought they would go real time.    Instead of me waxing on about real time let me pass the baton over to Om — he summarizes the shift as well as one could:

1. “The web is transitioning from mere interactivity to a more dynamic, real-time web where read-write functions are heading towards balanced synchronicity. The real-time web, as I have argued in the past, is the next logical step in the Internet’s evolution. (read)
2. The complete disaggregation of the web in parallel with the slow decline of the destination web. (read)
3. More and more people are publishing more and more “social objects” and sharing them online. That data deluge is creating a new kind of search opportunity. (read)”

Only connect …

The social aspects of this real time stream are clearly a core and emerging property.   Real time gives this ambient stream a degree of connectedness that other online media types haven’t.  Presence, chat, IRC and instant messaging all gave us glimmers of what was to come but the “one to one” nature of IM meant that we could never truly experience its social value.    It was thrilling to know someone else was on the network at the same time as you — and very useful to be able to message them but it was one to one.    Similarly IRC and chats rooms were open to one to many and many to many communications but they usually weren’t public.   And in instances that they were public the tools to moderate and manage the network of interactions were missing or crude.   In contrast the connectedness or density of real time social interactions emerging today is astounding — as the examples in the collage above illustrate.    Yet its early days.    There are a host of interesting questions on the social front.    One of the most interesting is, I think, how willthe different activity streams intersect and combine / recombine or will they simple compete with one another?      The two dominant, semi-public, activity streams today are Facebook and Twitter.    It is easy to think about them as similar and bound for head on competition — yet the structure of these two networks is fairly different.    Whether its possible or desirable to combine these streams is an emerging question — I suspect the answer is that over time they will merge but its worth thinking about the differences when thinking about ways to bring them together.      The key difference I observe between them are:

#1. Friending on Facebook is symmetrical — on Twitter it’s asymmetrical.    On Facebook if I follow you, you need to follow me, not so on Twitter, on Twitter I can follow you and you can never notice or care.   Similarly, I can unfollow you and again you may never notice or care.   This is an important difference.   When I ran Fotolog I observed the dynamics associated with an asymmetrical friend network — it is, I think, a closer approximation of the way human beings manage social relationships.    And I wonder the extent to which the Facebook sysmetrical friend network was / is product of the audience for which Facebook was intially created (students).   When I was a student I was happy to have a symmetrical social network, today not so much.

#2. The data on Facebook is assumed to be mostly private, or shared within private groups, Facebook itself has been mostly closed to the open web — and Facebook asserts a level of ownership over the data that passes through its network.   In contrast the data on Twitter is assumed to be public and Twitter asserts very few rights over the underlying data.    These are broad statements — worth unpacking a bit.    Facebook has been called a walled garden — there are real advantages to a walled garden — AOL certainly benefited from been closed to the web for a long long time.   Yet the by product of a closed system is that (a) data is not accessible or searchable by the web in general –ie: you need to be inside the garden to navigate it  (b) it assumes that the pace innovation inside the garden will match or exceed the rate of innovation outside of the garden and (c) the assertion of rights over the content within the garden means you have to mediate access and rights if and when those assets flow out of the garden.   Twitter takes a different approach.     The core of Twitter is a simple transport for the flow of data — the media associated with the post is not placed inline — so Twitter doesnt need to assert rights over it.    Example — if I post a picture within Facebook, Facebook asserts ownership rights over that picture, they can reuse that picture as they see fit.    If i leave Facebook they still have rights to use the image I posted.    In contrast if I post a picture within Twitter the picture is hosted on which ever service I decided to use.   What appears in Twitter is a simple link to that image.   I as the creator of that image can decide whether I want those rights to be broad or narrow.

#3. Defined use case vs. open use case.    Facebook is a fantastically well designed set of work-flows or use cases.   I arrive on the site and it present me with a myriad of possible paths I can follow to find people, share and post items and receive /measure associated feedback. Yet the paths are defined for the users.   If Facebook  is the well organized, pre planned town Twitter is more like new urban-ism — its organic and the paths are formed by the users.    Twitter is dead simple and the associated work-flows aren’t defined, I can devise them for myself (@replies, RT, hashtags all arose out of user behavior rather than a predefined UI.   At Fotolog we had a similar set of emergent, user driven features.  ie:  groups formed organically and then over time the company integrated the now defined work-flow into the system).    There are people who will swear Twitter is a communications platform, like email or IM — other say its micro-blogging — others say its broadcast — and the answer is that its all of the above and more.   Its work flows are open available to be defined by users and developers alike.   Form and content are separated in way that makes work-flows, or use cases open to interpretation and needs.

As I write this post Facebook is rapidly re-inventing itself on all three of the dimensions above.    It is changing at a pace that is remarkable for a company with its size membership.     I think its changing because Facebook have understood that they cant attempt to control the stream — they need to turn themselves inside out and become part of the web stream.   The next couple of years are going to be pretty interesting.       Maybe E.M. Forrester had it nailed in Howard’s End:  “Only connect! That was the whole of her sermon  … Live in fragments no longer.”

The streams are open and distributed and context is vital

The streams of data that constitute this now web are open, distributed, often appropriated, sometimes filtered, sometimes curated but often raw.     The streams make up a composite view of communications and media — one that is almost collage like (see composite media and wholes vs. centers).     To varying degrees the streams are open to search / navigation tools and its very often long, long tail stuff.  Let me run out some data as an example.     I pulled a day of bit.ly data — all the bit.ly links that were clicked on May 6th.      The 50 most popular links  generated only 4.4% (647,538) of the total number of clicks.    The top 10 URL’s were responsible for half (2%) of those 647,538 clicks.  50% of the total clicks (14m) went to links that received  48 clicks or less.   A full 37% of the links that day received only 1 click.   This is a very very long and flat tail — its more like a pancake.   I see this as a very healthy data set that is emerging.

Weeding out context out of this stream of data is vital.     Today context is provided mostly via social interactions and gestures.    People send out a message — with some context in the message itself and then the network picks up from there.   The message is often re-tweeted, favorite’d,  liked or re-blogged, its appropriated usually with attribution to creator or the source message — sometimes its categorized with a tag of some form and then curation occurs around that tag — and all this time, around it spins picking up velocity and more context as it swirls.    Over time  tools will emerge to provide real context to these pile up’s.   Semantic extraction services like Calais, Freebase, Zemanta, Glue, kynetx and Twine will offer a windows of context into the stream — as will better trending and search tools.      I believe search gets redefined in this world, as it collides with navigation– I blogged at length on the subject last winter.   And filtering  becomes a critical part of this puzzle.   Friendfeed is doing fascinating things with filters — allowing you to navigate and search in ways that a year ago could never have been imagined.

Think chunk
Traffic isnt distributed evenly in this new world.      All of a sudden crowds can show up on your site.     This breaks with the stream metaphor a little — its easy to think of flows in the stream as steady — but you have to think in bursts — this is where words like swarms become appropriate.    Some data to illustrate this shift.   The charts below are tracking the number of users simultaneously on a site.    The site is a political blog.    You can see on the left that the daily traffic flows are fairly predictable — peaking around 40-60 users on the site on an average day, peaks are around mid day.    Weekends are slow  — the chart is tracking Monday to Monday, from them wednesday seems to be the strongest day of the week — at least it was last week.   But then take a look at the chart on the right — tracking the same data for the last 30 days.   You can see that on four occasions over the last 30 days all of a sudden the traffic was more than 10x the norm.   Digging into these spikes — they were either driven by a pile up on Twitter, Facebook, Digg or a feature on one of the blog aggregation sites.    What do you do when out of no where 1000 people show up on your site?

The other week I was sitting in NY on 14th street and 9th Avenue with a colleague talking about this stuff.   We were accross the street from the Apple store and it struck me that there was a perfect example of a service that was setup to respond to chunky traffic.     If 5,000 people show up at an Apple store in the next 10 minutes — they know what to do.   It may not be perfect but they manage the flow of people in and out of the store, start a line outside, bring people standing outside water as they wait. maybe take names so people can leave and come back.   I’ve experienced all of the above while waiting in line at that store.   Apple has figured out how to manage swarms like a museum or public event would.    Most businesses and web sites have no idea how to do this.    Traffic in the other iterations of the web was more or less smooth but the future isnt smooth — its chunky.    So what to do when a burst takes place?   I have no real idea whats going to emerge here but cursory thoughts include making sure the author is present to manage comments etc., build in a dynamic mechanism to alert the crowd to other related items?    Beyond that its not clear to me but I think its a question that will be answered — since users are asking it.    Where we are starting at betaworks is making sure the tools are in place to at least find out if a swarm has shown up on your site.    The example above was tracked using Chartbeat — a service we developed.    We dont know what to do yet — but we do know that the first step is making sure you actually know that the tree fell — real time.

Where is Clementis’s hat? Where is the history?

I love that quote from Kundera.    The activity streams that are emerging online are all these shards — these ambient shards of people’s lives.    How do we map these shards to form and retain a sense of history?     Like the hat objects exist and ebb and flow with or without context.    The burden to construct and make sense of all of this information flow is placed, today, mostly on people.    In contrast to an authoritarian state eliminating history — today history is disappearing given a deluge of flow, a lack of tools to navigate and provide context about the past.    The cacophony of the crowd erases the past and affirms the present.   It started with search and now its accelerated with the now web.    I dont know where it leads but I almost want a remember button — like the like or favorite.   Something that registers  something as a memory — as an salient fact that I for one can draw out of the stream at a later time.   Its strangely compforting to know everything is out there but with little sense of priority of ability to find it it becomes like a mythical library — its there but we cant access it.

Unfinished

This media is unfinished, it evolves, it doesnt get finished or completed.    Take the two quotes below — both from Brian Eno, but fifteen years apart — they outline some of the boundaries of this aspect of the stream.

In a blinding flash of inspiration, the other day I realized that “interactive” anything is the wrong word. Interactive makes you imagine people sitting with their hands on controls, some kind of gamelike thing. The right word is “unfinished.” Think of cultural products, or art works, or the people who use them even, as being unfinished. Permanently unfinished. We come from a cultural heritage that says things have a “nature,” and that this nature is fixed and describable. We find more and more that this idea is insupportable – the “nature” of something is not by any means singular, and depends on where and when you find it, and what you want it for. The functional identity of things is a product of our interaction with them. And our own identities are products of our interaction with everything else. Now a lot of cultures far more “primitive” than ours take this entirely for granted – surely it is the whole basis of animism that the universe is a living, changing, changeable place. Does this make clearer why I welcome that African thing? It’s not nostalgia or admiration of the exotic – it’s saying, Here is a bundle of ideas that we would do well to learn from.  (Eno, Wired interview, 1995)

In an age of digital perfectability, it takes quite a lot of courage to say, “Leave it alone” and, if you do decide to make changes, [it takes] quite a lot of judgment to know at which point you stop. A lot of technology offers you the chance to make everything completely, wonderfully perfect, and thus to take out whatever residue of human life there was in the work to start with. It would be as though someone approached Cezanne and said, “You know, if you used Photoshop you could get rid of all those annoying brush marks and just have really nice, flat color surfaces.” It’s a misunderstanding to think that the traces of human activity — brushstrokes, tuning drift, arrhythmia — are not part of the work. They are the fundamental texture of the work, the fine grain of it. (Eno, Wired interview, 2008)

The media, these messages, stream — is clearly unfinished and constantly evolving as this post will likely also evolve as we learn more about the now web and the emerging social distribution networks.

Addendum, some new links

First — thank you to Alley Insider for re-posting the essay, and to TechCrunch and GigaOm for extending the discussion.    This piece at its heart is all about re-syndication and appropriation – as Om said “its all very meta to see this happen to the essay itself”.     There is also an article that I read after posting from Nova Spivack that I should have read in advance — he digs deep into the metaphor of the web as a stream.    And Fred Wilson and I did a session at the social media bootcamp last week where he talked about shifts in distribution dynamics — he outlines his thoughts about the emerging social stack here.   I do wish there was an easy way to thread all the comments from these different sites into the discussion here — the fragmentation is frustrating, the tools need to get smarter and make it easier to collate comments.

]]> http://www.borthwick.com/weblog/2009/05/13/699/feed/ 101 http://www.borthwick.com/weblog/2009/02/17/bitly-now/ http://www.borthwick.com/weblog/2009/02/17/bitly-now/#comments Tue, 17 Feb 2009 23:12:04 +0000 John http://www.borthwick.com/weblog/?p=629 We have had a lot going on at bit.ly over the past few weeks — some highlights — starting with some data.

• bit.ly is now encoding (creating) over 10m URL’s or links a week now — not too shabby for a company that was started last July.

• We picked the winners of the API contest last week after some excellent submissions

• Also last week the bit.ly team started to push out the new real time metrics system. This system offers the ability to watch in real time clicks to a particular bit.ly URL or link  The team are still tuning and adjusting the user experience but let me outline how it works.

If you take any bit.ly link and add a “+” to the end of the URL you get the Info Page for that link.  Once you are on the info page you can see the clicks to that particular link updated by week, by day or live — a real time stream of the data flow.

An example:

On the 15th of February a bit.ly user shortened a link to an article on The Consumerist about Facebook changing their terms of service.  The article was sent around a set of social networks and via email with the following link http://bit.ly/mDwWb.   It picked up velocity and two days later the bit.ly info page indicates that the link has been clicked on over 40,000 times — you can see the info page for this link below (or at http://bit.ly/mDwWb+ ).

In the screenshot below

1.) you see a thumbnail image of the page, its title, the source URL and the bit.ly URL.    You also see the total number of clicks to that page via bit.ly, the geographical distribution of those clicks, conversations about this link on Twitter, FriendFeed etc and the names of other bit.ly users who shortened the same link.

2.) you see the click data arrayed over time.:

The view selected in the screenshot above is for the past day — in the video below you can see the live data coming in while the social distribution of this page was peaking yesterday.

This exposes intentionality of sharing in its rawest form.   People are taking this page and re-distributing it to their friends.     The article from the Consumerist is also on Digg — 5800 people found this story interesting enough to Digg it.   Yet more than 40,000 people actually shared this story and drove a click through to the item they shared.     bit.ly is proving to be an interesting complement to the thumbs up.   We also pushed out a Twitter bot last week that publishes the most popular link on bit.ly each hour.    The content is pretty interesting.   Take a look and tell me what you think — twitter user name: bitlynow.

————–

A brief note re: Dave Winer’s post today on on bit.ly.

Dave is moving on from his day to day involvement with bit.ly — I want to thank him for his ideas, help and participation.     It was an amazing experience working with Dave.    Dave doesnt pull any punches — he requires you to think — his perspective is grounded in a deep appreciation for practice — the act of using products — understanding workflow and intuiting needs from that understanding.   I learnt a lot.     From bit.ly and from from me — thank you.

A pleasure and a privildege.