Ongoing tracking of the real time web …

The last post that I did about real time web data mixed data with a commentary and a fake headline about how data is sometimes misunderstood in regards to the real time web.    This post repeats some of that data but the focus of the post is the data.   I will update the post periodically with relevant data that we see at betaworks or that others share with us.   To that end this post is done in reverse order with the newest data on top.

Tracking the real time web data

The measurement tools we have still only sometimes work for counting traffic to web pages and they certainly dont track or measure traffic in streams let alone aggregate up the underlying ecosystems that are emerging around these new markets.  At betaworks we spend a lot of time looking at and tracking this underlying data set.   It’s our business and its fascinating.   Like many companies each of the individual businesses at betaworks have fragments of data sets but because betaworks acts as ecosystem of companies we can mix and match the data to get results that are more interesting and hopefully offer greater insight

——————————-

(i) tumblr growth for the last half of 2009

Another data point re: growth of the real time web through the second half of last year through to Jan 18th of this year.  tumblr continues to kill it.     I read this interesting post yesterday about how tumblr is leading in its  category through innovation and simple, effective, product design.   The compete numbers quoted in that post are less impressive than these directly measured quantcast numbers.


(h) Twitter vs. the Twitter Ecosystem

Fred Wilson’s post adds some solid directional data on the question of the size of the ecosystem.   “You can talk about Twitter.com and then you can talk about the Twitter ecosystem. One is a web site. The other is a fundamental part of the Internet infrastructure. And the latter is 3-5x bigger than the former and that delta is likely to grow even larger.”

(g) Some early 2010 data points re: the Real Time Web

  • Twitter: Jan 11th was the highest usage day ever (source: @ev via techcrunch)
  • Tweetdeck: did 4,143,687 updates on Jan 8, yep 4m. Or, 48 per second (source: Iain Dodsworth / tweetdeck internal data)
  • Foursquare: Jan 9th biggest day ever.    1 update or check-in per second (source: twitter and techcrunch)
  • Daily Booth: in past 30 days more than 10mm uniques (source: dailybooth internal data)
  • bit.ly: last week was the largest week ever for clicks on bit.ly links. 564m were clicked on in total. On the Jan 6th there were a record of 98m decodes.    1100 clicks every second.

(f) Comparing the real time web vs. Google for the second half of 2009

Andrew Parker commented on the last post that the chart displaying the growth trends was hard to decipher and that it maybe simpler to show month over month trending.  It turns out the that month over month is also hard to decipher.   What is easier to read is this summary chart.    It shows the average month over month growth rates for the RT web sites (the average from Chart A).   Note 27.33% is the average growth rate for the real time web companies in 2009 — that’s astounding.    The comparable number for the second half of 2009 was 10.5% a month — significantly lower but still a very big number for m/m growth.

(e) Ongoing growth of the real time stream in the second half of 2009

This is a question people have asked me repeatedly in the past few weeks.  Did the real time stream grow in Q4 2009?    It did.    Not at the pace that it grew during q1-q3, but our data at betaworks confirms continued growth.   One of the best proxies we use for directional trending in the real time web are the bit.ly decodes.   This is the raw number of bit.ly links that are clicked on across the web.    Many of these clicks occur within the Twitter ecosystem, but a large number are outside of Twitter, by people and by machines — there is a surprising amount of diversity within the real time stream as I posted about a while back.

Two charts are displayed below.    On the bottom are bit.ly decodes (blue) and encodes (red)  running through the second half of last year.    On the top is a different but related metric.   Another betaworks company is Twitterfeed.    Twitterfeed is the leading platform enabling publishers to post from their sites into Twitter and Facebook.    This chart graphs the total number of feeds processed (blue) and the total number of publishers using Twitterfeed, again through the second half of the year (note if the charts inline are too small to read you can click though and see full size versions).   As you can see similar the left hand chart — at Twitterfeed the growth was strong for the entire second half of 2009.

Both these charts illustrate the ongoing shift that is taking place in terms of how people use the real time web for navigation, search and discovery.    My preference is to look at real user interactions as strong indicators of user behavior.   For example I actually find Google trends more useful often than comScore, Compete or the other “page” based measurement services.   As interactions online shift to streams we are going to have to figure out how measurement works. I feel like today we are back to the early days of the web when people talked about “hits” — it’s hard to parse the relevant data from the noise.  The indicators we see suggest that the speed at which this shift to the real time web is taking place is astounding.   Yet it is happening in a fashion that I have seen a couple of times before.

(d) An illustration of the step nature of social growth. bit.ly weekly decodes for the second half of 2009.

Most social networks I have worked with have grown in a step function manner.  You see this clearly when you zoom into the bit.ly data set and look at weekly decodes, illustrated above.   You often have to zoom in and out of the data set to see and find the steps but they are usually there.     Sometimes they run for months — either up or sideways.    You can see the steps in Facebook growth in 2009.    I saw effect up close with ICQ, AIM, Fotolog, Summize and now with bit.ly.   Someone smarter than me has surely figured out why these steps occur.    My hypothesis is that as social networks grow they jump in a sporadic fashion from one dense cluster of relationships to a new one.   The upward trajectory is the adoption cycle of that new, dense cluster and the flat part of the step is the period between the step to next cluster.     Blended in here there are clearly issues of engagement vs. trial.   But it’s hard to weed those out from this data set.   As someone mentioned to me in regards to the last post this is a property of scale-free networks.

(c) Google and Amazon in 2009

Google and Amazon — this is what it looked like in 2009:

It’s basically flat.     Pretty much every user in the domestic US is on Google for search and navigation and on Amazon for commerce — impressive baseline numbers but flat for the year (source: Quantcast).  So then lets turn to Twitter.

(b) Twitter – an estimate of Twitter.com and the Twitter ecosystem

Much ink has been spilt over Twitter.com’s growth in the second half of the year.   During the first half of the year Twitter’s experience hyper growth — and unprecedented media attention.    In the second half of the year the media waned, the service went through what I suspect was a digestion phase — that step again?     Steps aside — because I dont in anyway seek to represent Twitter Inc. — there are two questions that in my mind haven’t been answered fully:

(i) what international growth in the second half of 2009?, that was clearly a driver for Facebook in ’09.  Recent data suggests growth continued to be strong.

(ii) what about the ecosystem.

Unsurprisingly its the second question that interests me the most.    So what about that ecosystem?    We know that approx 50% of the interactions with the Twitter API occur outside of Twitter.com but many of those aren’t end user interactions.     We also know that as people adopt and build a following on Twitter they often move up to use one of the client or vertical specifics applications to suit their “power” needs.   At TweetDeck we did a survey of our users this past summer.     The data we got suggested 92% of them then use Tweetdeck everyday — 51% use Twitter more frequently since they started using TweetDeck.  So we know there is a very engaged audience on the clients.     We also know that most of the clients arent web pages — they are flash, AIR, coco, iPhone app’s etc. all things that the traditional measurement companies dont track.

What I did to estimate the relative growth of the Twitter ecosystem is the following.   I used Google Trends and compiled data for Twitter and the key clients.    I then scaled that chart over the Twitter.com traffic.   Is it correct? — no.   Is it made up? — no.   It’s a proxy and this is what it looks like (again, you can click the chart to see a larger version).

Similar to the Twitter.com traffic you see the flattening out of the ecosystem in the summer.    But you see growth in the forth quarter that returns to the summer time levels.     I suspect if you could zoom in and out of this the way I did above you would see those steps again.

(a) The Real Time Web in 2009

Add in Facebook (blue) and Meebo (green) both steaming ahead — Meebo had a very strong end of year.    And then tile on top the bit.ly data and the Twitterfeed numbers (bit.ly on the right hand scale) and you have an overall picture of growth of the real time web vs. Google and Amazon.   As t

  • http://yallaguy.com aarondelcohen

    Useful John. When does the real-time economic engine start growing to match the audience acquisition? Put another way, what marketplace can be created to engage google's advertisers or amazon's vendors to begin shifting or at least experimenting with their marketing dollars in way that is as measurable as Google's. My concern about google is less the overwhelming reach and day to day position they hold in so many consumers' lives than the hegemony they have created in the various marketing industries. In particular, I'm thinking about the growing trend to link marketing to sales.

    • Johnborthwick

      Great question. I dont think the marketplace has emerged as yet. I have heard anecdotally that spend is moving to Facebook's ad platform — in particular advertisers who want to reach customers pre intent — ie; before the are ready to search and find a particular product. Its early days and my belief is that the tools we used on the previous platforms wont necessarily port over — new stuff needs to be built.

    • http://www.cheap-lacostepoloshirts.co.uk lacoste polo

      Very optimistic. A large fraction of the exact pscyhographic who would ask, “Did the real time stream grow in Q4 2009?” missed your point.

  • Anonymous

    Thanks very much for this insightful (and very unique) analysis, John! I especially like the “bit.ly weekly decodes” information you provide; I believe this illustrates a steady increase in the value-added use of Twitter, i.e. users injecting valued resources into their Twitter streams.

    Another indicator of such value-added Tweeting are hashtags; it would be interesting to layer the http://hashtags.org analysis of trends onto yours!

  • http://rafer.tumblr.com rafer

    John,

    You can probably check devkey usage to explain the step function growth in weekly decodes. Bitly isn't unto itself social enough to explain the pops directly. It's highly useful to very social services, which should.

    And, using a log scale without explicitly calling it out and linking to a how-to-read-this-graph page? Very optimistic. A large fraction of the exact pscyhographic who would ask, “Did the real time stream grow in Q4 2009?” missed your point.

    • Johnborthwick

      Scott – re: the log scale. FYI this is what it looks like without the y axis on a log scale – http://bit.ly/7MRUdi – same conclusion but the log scale reduces the variability of the RT web line.

      • http://rafer.tumblr.com rafer

        Cool. It appears that the non-tech population have an easier time rationalizing that up and to the right can be bumpy than they do understanding log scale.

  • dailyboothtrafficquestions

    Please stop recycling the 10M number for dailybooth unless they give you third-party proof. Look at compete (150K uniques), comScore (200K uniques), quantcast, or alexa. Nothing bears it out. They have less than 500K uniques in the US by any valid measure. Either their traffic is heavy international and US is no where near their top country or they are doing something strange in their counting. I could make my Google Analytics number say anything I want. That's why no one quotes that number with a straight face — except DailyBooth.

    Ask him to prove the number to you with a third party stat. I believe it's possible they have an order of magnitudes less traffic than they claim. In any event, they never say much more than complain about third-party tracking, which is bunk. Those tracking services are directionally accurate and generally pretty good. There's no question they can be off, but not for so long and by more than an order of magnitude.

    • Johnborthwick

      Cranky DailyBooth Person. I'm looking at the Dailybooth numbers for the past 30 days and the number is 11.09M. As I said at the outset — measurement of web pages (let alone API client / based services) still has a considerable amount of variance. In particular for flash, widget based sites. This is a solid article about how bad this stuff can get: But Who's Counting? http://bit.ly/4ErSgk

  • http://twitter.com/teasastips LaTease Rikard

    It sounds like real-time tracking is more reliable and in tune with real people. Is organic search running behind? It looks like it.

  • Pingback: Bit.ly and the Twitter Ecosystem | Venture Capital

  • http://www.feedtrace.com Bernardo Rodriguez

    As value added apps continue to emerge, the twitter ecosystem should grow orders of magnitude above twitter.com. We look at twitter as a large real-time database with metadata exposed through URLs. Apps will cache data and add value. The fact that this database includes user ID and social graph gives an unprecedented opportunity. Today most of twitter interaction is around personal communication, in the next years we should see exponential growth in search, navigation and recommendation apps that work over tweet’s metadata.

  • http://portal.eqentia.com William Mougayar

    Great analysis. It could be argued that Twitter+ecosystem are a (real-time) proxy to web content (except for originally generated user content). What's not clear yet is whether the 'real-time' part of the web will continue to look like the fast & furious segment of the web, or whether its value will get integrated across the web & mobile web.

  • Pingback: Twitter’s Platform Mojo « leaving the flock

  • http://twitter.com/ocoudert Olivier Coudert

    Very interesting data, thanks for sharing. I agree with the claim that the twitter eco-system as a whole (including twitter.com) is larger than twitter.com itself –at least x3 larger, and up to x5 larger if you restrict the analysis to active users, see http://thenextweb.com/2010/01/12/twitter-flattening-short-answer/.

    Still, Twitter must do better to attract a new wave of users, and keep them engaged. I would like to see some more innovation and integration. Having an a few 1000′s of companies in the eco-system is great, but it may be time to consolidate some of these services into a easy-to-use, comprehensive, platform: read, publish, search, cross-reference.

  • Pingback: Demand-Side Platform Adchemy Opens Kimono; Collective Offering DSP-like Video Buying; Ghostery Bought By Better Advertising

  • Pingback: Twitter.com Vs. The Twitter Ecosystem | TechBlogs Today

  • Pingback: Is Twitter Tanking? An Update « Jeffbullas’s Blog

  • Pingback: Twitter Thinks It Can Evade Chinese Censorship – GigaOM

  • Pingback: Twitter Thinks It Can Evade Chinese Censorship | Robust Tech

  • bonnwafer

    Just curious on the mobile side share. Smart phones are now much more affordable and equipped with social networking features.

  • http://www.titidirectonline.co.uk/ski-snowboard-goggles skiing goggles

    great stuff

  • http://www.titidirectonline.co.uk/ski-snowboard-goggles skiing goggles

    great stuff

  • Pingback: So many people, So many tweets, So what? | digiAindra - One-stop digimarketing

  • http://www.atkins-diet.cc/ Atkins

    Nice research. Thanks for the data.

  • http://www.louisvuittonhouse.com/ louis vuitton handbags

    You can probably check devkey usage to explain the step function growth in weekly decodes. Bitly isn't unto itself social enough to explain the pops directly. It's highly useful to very social services, which should.

  • http://www.cheap-ralphlaurenpoloshirts.com big pony polo

    Great analysis

  • http://www.guccishoesmenss.com gucci boots

    The atmosphere was primed and all it needed was a trigger (southerly change coming in earlier than expected!) and plenty of moisture. James and Dann also chase today and should have some more info about this storm from the NW suburbs. It was great fun! plenty of development, some very heavy rain and strong winds as well as 1cm hail.

  • Anonymous

    “the real time web”the information it’s very useful for me thank you very much!!!!