I’m sitting here with 3 LCD screens in front of me with database queries, spreadsheets and browser windows open trying to determine how accurate Mobilytics is.

Let’s forget about mobile for a minute and just talk about web analytics in general. Before factoring in the complexities of mobile, I wanted to get a reference point where I could say that Mobilytics was correctly tracking a regular Internet web site.

Since we are not using JavaScript, this would be a great test of the accuracy of our server side snippet code in general.

So I decided to compare Mobilytics to a few web analytics products out there which I will not name. I will say that one of them is Google Analytics though. The problem with Google is that I can’t drill down to individual visits to compare page view by page view.

So here’s what I came up with. NO WEB ANALYTICS PRODUCT WILL EVER BE RIGHT!

It’s just not possible. There are too many opportunities for minor issues to impact the data. The real question is, how would you know if it was right anyway? And how good is good enough?

Here I am on a Sunday morning comparing my raw log files and database queries side by side with 2 other web analytics products. I placed each product’s tracking code on the same Internet web site, so you would expect them to match up.

Not the case. And I’m not just talking about general numbers and percentages. I am auditing individual page views one at a time. While I can understand the occasional duplicate appearing, or the time on the page being off,  how is it that I have visitors in my data that don’t show up at all in this other vendor’s product?  The one particular visit that led me to blog about this is coming from the U.S. on CabelVision using Firefox. So why would it not be appearing in the other product?

One thing I can think of is that Web Analytics rely on JavaScript and Cookies. Maybe this visitor turned them off? Well since we are using server side tracking code, we have the complete HTTP headers and other info so I can easily check. (doing this now), be right back…

Ok so here’s what I found. Clear evidence of a visitor who came through a referral link on a blog, and is not a crawler or search bot of any type. While I can’t tell in this case if they have cookies turned off, you would think with Firefox that would not be the case. But let’s assume it is. Wouldn’t we at least register a visit?  Not only did I lose a visitor, I lost a referrer, a page view, and I would never have even known. What if I am an advertiser and I am paying to attract visitors? Is it ok for one to be completely missing?

Who else is reviewing logs and comparing vendors to see how accurate they are? Now I wonder  how often this happens? I’m only reviewing a 1  hour period on a single day.

On mobile we can’t rely on cookies and Javascript, why does the wired web take them for granted? I guess the same way we need to discard data sometimes in order to maintain the integrity of the data, the web vendors do also. It just seems to me that there is no reason you can’t use an image tag and Javascript.

And just to be clear, I don’t even know if this is the reason the visitor was missing. I will be looking further though.

So the simple conclusion I have drawn from hours of comparisons in general is that putting the numbers side by side will NEVER match up exactly. At some point they will vary. But there’s no real way the average user would know anyway.

I’m looking forward to putting a few of the mobile products side by side and seeing how we stack up. The quality and accuracy of the reporting is the single most important thing we are focusing on, and that is what will make Mobilytics the leading third party analytics and tracking solution for mobile content. Why else would I be comparing single page views and HTTP headers on a beautiful Spring Sunday?

I finally got a chance to read through a post on the Bango blog by their CEO Ray Anderson regarding the still somewhat unknown Novarra transcoding SNAFU. Ray has commented on an interview at Mobile Marketing Magazine with Novarra’s COO.

Ms Rangarajan, as COO of Novarra is at best badly mis-informed or at worst blatantly transcoding the truth in her interview with the Magazine.

I completely agree with Ray on this one.

For those of you not aware of this issue, here’s what’s going on. There are a number of mobile carriers using Novarra’s transcoding technology which allows ordinary Internet web sites to be adapted for mobile handsets. They are not the first transcoder on the web, but I believe they are the first that completely hides the information about the actual phone and browser it is being viewed on.

When Vodafone UK launched the Novarra transcoder on the now infamous 7/7/07 , to enable desktop optimised websites to render better on mobile, the result was outcry from thousands of content providers large and small that their sites could no longer be seen, were corrupted, or were destroyed.

http://uk.techcrunch.com/2007/09/21/vodafone-in-mobile-web-storm/

Advertisers on mobile sites found their ads were no longer viewed. People selling mobile content had a surge in customer complaints - content that was sold did not work (the phone type was unknown) or simply not delivered.

Vodafone UK had not forseen the problem, but reacted very quickly to create a “whitelist” of sites that would bypass the transcoder - and this patch is still in place. Organizations like the BBC, CNN, SKY, Facebook, Flickr and others had service restored within a few days. The problem remains however that sites are “adapted” or corrupted unless the website owner knows how to get themselves added to the whitelist. Estimates of the financial losses to website owners in the UK range from hundreds of thousands of pounds upwards.

Those of us that are techies know that Novarra had to go out of their way to hide the original user agent information. There is no question in my mind that this was done intentionally. While we can identify the visitors from Novarra gateways with Mobilytics, we do not have any info on the phone model, capabilities or other identifying information.

According to Novarra’s COO:

“The problem only arose on mobile sites that were not .wap or .mobi. - but there were a lot of people who didn’t address their mobile sites as .wap or .mobi. They addressed them as .com, because in the world they were used to, it wasn’t possible for their regular website to be rendered on a phone. “

Ray’s reply:

Er, yes. I have no idea what a “.wap” site is, but of course most sites will be .com, .net, .de or .co.uk or whatever. They will be in peoples bookmarks, sent to users in messages etc., why should a person have to change their site address to stop Novarra breaking their site?

This is crazy! They are transcoding mobile web sites that are already built to render perfectly on mobile phone. Mobile developers have painstakingly built these sites to render properly depending on what size screen a phone has and it’s capabilities. Novarra is completely stripping that information. At a minimum all they need to do is look at the headers and not transcode a site that has the proper doc type. They claim to not transcode .mobi or .wap (I’ve never heard of this one either Ray.), but that’s not good enough.

While Novarra would like to claim that one very loud voice is causing industry backlash, this is simply not true. Also, without the Analytics that Ray and I provide, many content providers probably don’t even know this is happening!

This issue needs to be publicized outside the tech community. Big brands and small site alike who have spent time and money tailoring their content need to speak up and let the Carriers that use Novarra’s technology know what’s going on. We are not asking them to stop rendering Internet Web Sites on Mobile, we are asking them the stop rendering Mobile Web Sites as Internet Web Sites for Mobile.

If we can write code to identify thousands of phones, why can’t the transcoders simply not transcode mobile web sites? If the site is .mobi compliant, then don’t transcode it! It’s that simple.

Better yet, here’s a simple fix that can be put in place in less than an hour. If you see this, add the site to the whitelist and don’t transcode it!

<!DOCTYPE html PUBLIC “-//WAPFORUM//DTD XHTML Mobile 1.0//EN”

"http://www.wapforum.org/DTD/xhtml-mobile10.dtd">

Even simpler is this: IF THE DOCTYPE CONTAINS THE WORD MOBILE THEN WHITELIST (Novarra, translate to whatever development language used. I put the pseudocode in the public domain.)

Well after letting it sink in for a few days, its time for some comments on the upcoming AdMob analytics.

While I am not in the habit of providing links to competing products, one of the things I said this blog would be about is mobile analytics. Not just Mobilytics.

We were aware that AdMob was coming out with something, but were led to believe that it was different than what was announced.

I want to share some comments I made on a blog post at GoMoNews.com on the subject. I think they are important.

Bena Roberts had asked if it was a smart move for AdMob to get involved in analytics.

Is AdMob going pear shaped? Ok. Ok. I have been accused of bullying AdMob in the past and it’s not my objective to bully – but I will sign up for BETA (if they let me) and report back. My concerns about this is that AdMob seems to be doing what is in fashion and not what’s essential for its ad network.

Yes, its easy to set up AdMob. But education for consumers is essential in the part of finding and choosing adverts for the site and writing killer short briefs. That is where there mobile ad networks are failing non-agency clients at the moment.

Just because AdMob has the data they think they need to do accurate analytics, doesn’t mean they should. Same goes for Bango. Yes, provide click tracking and ROI reporting, but why try, or claim to be a complete analytics solution? This is not what they are good at. Here are two leaders in their own niche that they both rule. If I was them I’d keep heading in that direction.

My guess is that AdMob believes that if they give away free analytics to every mobile website, that they will add more publishers and advertisers. We’ll see how that goes.

But realty says that no matter what medium ads are bought and sold on, there needs to be sufficient tracking and auditing to keep everyone honest and accountable. There needs to be somewhere reliable that you can use to track your performance and compare across ad spends. Buying ads from AdMob, and then using their analytics to verify is a conflict of interest if you ask me. Same goes for publishers. If I am getting paid for page views or clicks, then why would I want to use the company paying me to verify how many there are?

For years Google has been paying publishers without giving them details of how they came up with the amount. AdMob has also been as transparent as a brick wall. As an AdMob advertiser, I have seen clicks to my site that are more than questionable, with no way to verify, until now. Clicks from spiders, desktop computers, and other suspicious activity. Do I want to use AdMob analytics to verify their own numbers??

If I want to compare my Yahoo Mobile, DeckTrade and AdMob spends to see which is more effective, do I want to use AdMob analytics to do so? Talk about a conflict of interest!

Why is it you think that Google Analytics is free, yet Omniture and hundreds of other web analytics products still thrive??

Mobilytics has been built from the ground up to be an impartial analytics and campaign tracking system specifically for the mobile market. We didn’t bolt it onto another product that already had some data. The reason we are still in beta testing is that each day we painstakingly sifting through data every day to improve the accuracy of our stats, and will not release it until we are satisfied.

For example, we spent significant development time building a Blackberry specific module into our coding servers. So instead of every Blackberry showing as having a carrier of RIM and a country of Canada, we can now identify the real carrier and country a large percentage of the time. You won’t be seeing comments like this one left from Glenn Walker at Bango.

Interestingly, you accessed the link on a Blackberry using the default internet connection, rather than the specific WAP gateway, hence then unidentified entries. We are currently working on improving our detection and reporting of devices which, although mobile devices, access the internet directly -

This was in response to me blogging about how their screens were all showing unidentified handsets.

Since Bango bolted their analytics on, they are relying on carrier relationships to identify phones according to them. I visited through a corporate BES server the way 90% of them do. Well what happens if I access on my iPhone from a WiFi connection.

Look for Mobilytics to be a stand-alone complete mobile analytics package that spans across the mobile web, sms, in-game advertising, video, Bluetooth and mobile email.

Look for an API that integrates into Omniture and other analytics packages, and provides all analytics data to our customers.

Personally I’m not comfortable with the fox guarding the hen house, and I’m betting that companies spending millions on advertising are not either.

If you are involved in the mobile industry, and have not joined your local Mobile Monday group, RUN to their web site now and join. There are many chapters throughout the world, and you can get more info at the main Mobile Monday Global site.

This month the NY group is doing an event on Mobile Analytics & Social Search.

This month’s event, “Optimizing the Mobile Experience and Increasing Visibility with Social Search and Mobile Analytics” is sponsored by taptu.

Bryson Meunier, Resolution Media: To demonstrate how rapidly the mobile web and mobile search is developing these days, ten months ago David Harper said at the Mobile Monday NY dedicated to Mobile SEO:

“There is a real lack of analytics, as far as traffic or benchmarks. Many analytics tools don’t understand mobile traffic.”

Today, however, there are many analytics packages in place to optimize mobile campaigns, including Amethon, Bango, Mobilytics and Quattro Wireless. The fast-paced world of mobile search has also brought Taptu to the forefront in the last six months with their mobile social search engine, which many have called a serious competitor to Google in mobile search.

Fortunately, Mobile Monday NY is hosting a free event on April 28th to discuss the new opportunities available in mobile search and discovery since that Mobile SEO panel just ten months ago. How can marketers best leverage social search and mobile analytics to bring visibility to their brand today? Representatives from several companies active in mobile analytics and social search, including Amethon, Bango, ESPN, Quattro Wireless, Mobilytics, Resolution Media, TigTags, and taptu, will be on hand to discuss. | RSVP here

This should be a wonderful discussion for the industry considering that 99% of the mobile analytics vendors are on the panel. If you are in the NYC area, please join us.

When:
Monday, April 28 2008 at 7:00 PM (until 9:00 PM)

Where:
Samsung Experience
Time Warner Center – Shops at Columbus Circle
10 Columbus Circle, 3rd floor
New York City, New York 10022
(map)

At the intersection of Broadway, Eighth Avenue, Central Park South and
Central Park West.

Subways to Shops at Columbus Circle: A/C, 1/9, B/D
to 59th Street/Columbus Circle

Well this one works out well for us. Amazon has lowered the cost of data transferred out of their servers, and have kept all transfer between their services as free.

We are generating a tremendous volume of log files from all the data we are tracking. The nice thing is that we do ALL the processing within the Amazon cloud until the last possible moment when we download it to our Windows reporting servers. We then compress it and store on Amazon S3. By the time we download it, the data is 1/5th the size of the original. Cha Ching! Hey every penny counts… Now if we could just get them to support Windows servers and we’d never have to leave the cloud.

Thanks Amazon!!

One thing I have seen over the last few months is that what people who want to track mobile web site stats want is features and metrics we have, and what they initially think Bango has based on press releases and marketing materials.

After being publicly challenged to provide a beta account to Bango by Andrew Bovingdon of Bango on the Mobile Monday London Yahoo Group, I went to take another look and see if I was really missing the point. It appears not. He also challenged the accuracy of our stats, without a clue of how we collect or analyze them. With Andrew’s permission, I will post his public email and my public response on the blog.

But for now, back to seeing what I was missing.

Well, for starters, they have this wonderful PDF that you download with pictures, examples and info. The only problem is that the screens in the PDF don’t exist in my account. In fact they show the same screen as I have, and then they show another screen shot of it with a menu that does not exist.  Well there’s one thing I’m missing. We’ll get back to this…

Here’s what I did:

I logged in using Firefox and went to get my Bango Link so I could visit my test site with my phone. When I clicked on the “Test This Content Link” button I got this –>

OUCH! You want me to type that in on my phone?? What happened to sending it to me via SMS at least.

So after a minute of typing, I was redirected to my site successfully.

So I clicked on my Analytics Center button and got this:

I was not surprised it was showing ZERO page visits, as I assumed it is not real time. What I could not understand, was where were the links that were in the PDF?

They are clearly there in the PDF.

So I then thought that maybe it was a browser issue so I tried it in Internet Explorer. Sometime sites use ActiveX controls that don’t work in Firefox.

What I immediately found interesting, was that the IE version was showing my test visit. No missing menu items, but there was my visit.

So I went back to Firefox and refreshed the screen. Nope still showing ZERO. After closing the browser and logging back in, the visit appeared. But still no missing menu items.

So my next though was that maybe I need a higher package. I’m using the free version.

So I go to Manage My Account, and click on Upgrade Package

I get a drop down list of choices with no info on what they are.

But they do have a link there for more info on the packages.

So I click it, and this is what I get….

HA! This page isn’t redirecting properly! How Ironic is that??? Coming from the mobile web’s authority on redirecting links!

Yes, I actually laughed out loud.

Well in any case, I never found the missing menus. What I did find though is this cool screen that shows all the live visitors and hits to the site. After a few minutes if “Updating…” this is what I got. Apparently they are all using the same phone, are from the same country, use the same network, and visited at the same time… UNDEFINED!

You can click the screenshot to see it for yourself, although I am sure they will fix it as soon as they get the Google alert about this blog post.

I couldn’t make this stuff up even if I wanted to…

So as far as I can tell, Bango Analytics is one single screen that does not let you change the dates you want to view, and does not track pageviews or any other actionable information other than Bango sales and landing page clicks. Why don’t they just call it a mobile link tracking tool or something? Call it a mobile campaign manager.

Just stop calling it mobile web site analytics!

While it’s been quiet on the blogging front lately, it has been anything but quiet back here at the office. The developers have been banging out features and bug fixes, while business development is racking up an impressive list of potential and committed clients. We’ve had great interest from companies that range from the little guy with a wap site, to a few of the top 10 Internet giants. (Still waiting for Google to call…)

Beginning Monday, April 7th, we will open the next round of beta testing. Over the next few weeks we are looking to add at least an additional 1 million page views per day. We will be sending out invites to specific requesters based on when they signed up, as well as the technology used and size of the site. But don’t worry, we are looking for both large and small sites. We will probably add about 30 beta testers in this round over the next 2 weeks.

One of the other things we will be looking for is sites that are interested in using our service for free (and earn revenue) in exchange for showing a small text ad. We need to start testing the ad servers and targeting platform in the real world. Please let us know if you are interested in participating. There is more information about our mobile ad network here on our web site.

Over the last few months, we have been working hard on making our infrastructure and applications as scalable as possible. Without getting to technical in this post, we have built out an entire network of virtual computers, disk storage and monitoring applications using the Amazon Web Services platform. I figure if it’s good enough for them to run Amazon.com on, then it’s good enough for us.

When things settle down, I plan to put together a number of posts and a technical document outlining exactly what we have done. There are many out there in the tech community that can learn from this experience. I have been in technology for over 20 years, but never before have I integrated such a wide range of products and operating systems with such power, and with such ease.

For those of you non-techies, all you need to know is that with a single click of a mouse, we can add additional capacity. As the page views grow, so does the processing and data collection power. We recently heard a story about a competitor of ours who was having major issues with processing time and overloaded traffic. While moving over to the Amazon Elastic Computing Cloud set us back a few months, we won’t need to rebuild our apps and infrastructure when we become the #1 mobile web analytics solution available.

Our goal for the last phone of testing is for this scalability to be completely automated. We are almost there. The system can see how much data is coming in and waiting to be processed, and it can turn on and off “virtual” computers as needed. That’s cool! We don’t need to pay for computers that are not being utilized, and it will all be done automagically.

Keep an eye out for your beta invite, and please respond quickly if possible. If you can’t take advantage of your invite, then we’d like to send some out to others who are waiting. We have received hundreds of requests, but can’t flip them all on at once (although we probably can), until we load test further.

In an article recently from BusinessWeek, Olga Kharif points out the issues that advertisers are having with the transparency of mobile ad networks.

Advertisers are shying away from mobile phones until they get clearer data on whether promotions on the small screen pay off

We have 2 main objectives that will determine if Mobilytics is indeed a successful product. The first being a simple web analytics tool to get an overall detailed picture of visitors and customers. We take this for granted on the Internet, and there is no reason we can’t have the same and more on the mobile web.

The second objective is to allow advertisers and publisher to track what’s going on with their campaigns. The ad networks right now do not share enough information, and it is difficult for advertisers to track end to end promotions to sales and conversions. It is also difficult to compare side by side ad campaigns from different ad networks and publishers.

Complicating matters, what little data the wireless service providers do pass back to advertisers varies widely in terms of what they measure. Mobile-advertising networks, in turn, crunch the disparate data in different ways to gauge the audience response to ads. One ad network might report the number of phones that received an ad, while another might report how many users actually viewed the ad. The distinction is subtle, but important for advertisers. Without a consistent yardstick, it’s hard to compare the results of a campaign that ran through Yahoo’s ad-placement business with one placed by a rival network such as AdMob.

Mobilytics is being built to be a tool that can leave the ad network out of the decision as to what data to share with the advertiser. With proper campaign coding, detailed information to be used across publishers is easily possible.

We are that consistent yardstick that is needed in the industry. The only way to measure correctly will be with a third-party tool that provides common metrics across publishers.

In the end, ad agencies find themselves creating complex spreadsheets in a bid to reconcile the data from various campaigns. “They are doing more manual processing of data than strategic planning for their clients,” says Scott Ferris, a senior vice-president at Microsoft (MSFT), which has been testing software to help agencies compare mobile-ad data from different sources.

Building manual spreadsheets is not the answer. It still uses bad input data. Using Mobilytics and our API we can provide all the data needed to determine the success of mobile ad campaigns.

MobiAd news has a great post about the article here as well.

A few weeks a go I was contacted by Bryson Meunier who has a blog about mobile search engine optimization and natural search. He was putting together a post about mobile analytics and wanted to include Mobilytics.

He released that post yesterday, and we are very pleased with it. In the post he looks at the 4 mobile analytics vendors that are in the market today. Of the four, only 2 of us are really complete stand-alone analytics packages, and we are the only one that can be used by everyone from tiny budget-minded mobile webmasters, to the major mobile web portals.

From what I’ve seen, Mobile Vision’s Mobilytics is in a position to become the Google Analytics of the mobile web. Mobilytics is similar to Bango and GetMobile Analytics in that it’s a hosted solution, and like both Bango and Amethon in that it offers mobile marketers a wealth of data about how their mobile web sites are used. The difference, and the reason why Mobilytics might catch on faster than the others, I think, is that they offer free mobile web analytics to anyone willing to put a small text ad on their site, regardless of the number of page views. This is likely to appeal to small business owners with the branding flexibility to do this, and other mobile marketers who are not convinced enough of the need for mobile-specific analytics to increase their budgets for them just yet.

In terms of their product offering, Mobilytics is competitive with both Amethon and Bango in the sheer number of data metrics that they offer marketers and the accuracy of their data relative to traditional web analytics. Like GetMobile Analytics, tracking is done through a widget of code that captures information about the server

He also points out some of the issues that our competitors face.

Amethon:

Requires a dedicated server and is not for small to medium budget sites that are on shared hosting platforms. It could also be an issue for enormous sites that are using server farms and load balancing since it can become a bottleneck.

Don’t own a server? Don’t start salivating just yet. Amethon bills their Site Edition as being ideal “for large mobile content publishers ”, meaning smaller to mid-size companies without the mobile budget might find their solution less than ideal. Tracking for Amethon works by switching the traffic from a mobile analytics server to a web server and requires additional hardware installation, necessitating a certain level of commitment of resources that many small businesses might not be willing to make just yet.

If you want to be up and tracking your complete mobile web site and marketing campaigns in minutes for minimal or no cost, Amethon is not the solution for you.

GetMobile Mobilizer:

In my opinion, GetMobile is not really an analytics solution. Their product is a mobile web site builder with mobile stats built in. They also lack many of the tracking features and metrics needed to really understand your site and visitors.

As part of their mobile site creator they offer mobile analytics to give webmasters a better sense of what’s happening on their mobile sites.

The problem with GetMobile analytics at the moment is not accuracy, but rather the lack of actionable metrics with regard to the mobile search engines. For SEOs, there’s not much here currently that could aid them in benchmarking, tracking and optimizing a campaign.

Bango Analytics:

Well my opinion of Bango has been publicly stated. While Bryson mentions some of the issues with using Bango, he does not address the fact that it is not a complete solution. Simply put, Bango is not a stand-alone mobile analytics solution that can provide all of these metrics, and track mobile campaigns and SEO easily. They require you to redirect your visitors to a Bango URL on the way in. It does not work with natural organic search, and does not track pageviews, search engine keywords, content paths, and many other needed analytics.

Bango is a good solution for users of their e-commerce services who want to track conversions and get details about their customers. We do all that and more. And we don’t require you to redirect the URL.

One possible problem with Bango Analytics with regard to SEO is the way that mobile users are tracked. Currently Bango tracks mobile users by temporarily redirecting them to a Bango.net site, where data is collected on the handset, search terms, etc. before they’re sent back to the client site. This is done so quickly that it doesn’t affect the user experience, but it could affect the overall link popularity or the crawlability of the site. I simulated a spider crawl on several of Bango’s mobile clients, and it doesn’t seem to be an issue with regards to crawlability, but it’s something to look out for if you decide to select Bango Analytics as your mobile analytics provider.

Essentially what it comes down to is that Mobilytics is a complete solution that combines the best of all of these solutions in a single package. Bango’s complete system is covered in our Campaign Tracker module and  Amethon’s “on the wire” data capture is covered by our optional IIS and APACHE custom logging modules (more on this soon).

We are also launching a mobile web site hosting & web site builder solution that will provide server based, detailed, accurate stats without the need for page tagging. More on this to come as well.

We’re very excited about the future, and look forward to adding our next round of beta testers next week.

Nice job Bryson! Thanks!

First let me start by thanking those mobile web sites that are currently participating in our beta, and those waiting for their invites.

Beta testing has been a part of software development since the first software was created. Contrary to what many “Web 2.0″ companies think, beta testing has a purpose other than just being an excuse for having bugs.

While we can create mock data and test features in-house, there is nothing like real data and users to find the faults, stress the system, and solicit valuable feedback. This is the reason for beta testing.

About a week ago, we put the invites on hold while we began implementing a solution to an issue that I was uncomfortable with. I won’t get too technical in this update, but will do a blog post about the system architecture sometime soon.

As an entrepreneur, I find it extremely valuable to get my web site analytics data as close to real time as possible. Two of my goals when designing a mobile web analytics product were to deliver analytic data within hours rather than next day, and to be able to add hundreds of millions of pages views without affecting that time frame.

Soon after we added our first large beta tester, we realized that we had a scalability issue when processing the raw data we receive from our customer’s mobile web sites. While it worked great for smaller sites, many of the calculations and processes were taking way too long, and it was not scaling well. Adding more sites and pageviews added too much time to the processing of the data.

I immediately began to architect a better way to process that data, and it appears to have made a huge difference. While our reporting servers and web site are Microsoft ASP.NET and SQL Server based, we have moved our data collection and analytic processing to a Linux based platform, utilizing Amazon’s EC2 elastic computing cloud, S3 storage, and SQS web services.

A successful hosted web analytics product can have a database with billions of records, and will receive many millions of hits per day from all over the world. For those of you that are not technical, Amazon’s web services allow unlimited growth by providing “virtual” servers and unlimited storage. Adding a mobile web site to our service that gets 100 million page views per month is as simple as turning on a few additional virtual servers.

Virtual Servers can also be used for analytical data processing, and can be turned on and off as needed based on the load. If one of our customers all of a sudden gets a huge spike in traffic, we automatically turn on additional servers, and turn them off when they are under-utilized.

At this point our data collection process has been completely moved to Amazon. Additionally, we are in the final stages of development and testing of our new coding and import process.  This is the process that takes the raw data and moves it to our reporting servers for display in our interface and also makes it available through our web service. We have moved it from a C# .Net & SQL Server application, to a Java based app that can run on Amazon’s servers and can scale up as needed. Using Amazon’s SQS service, the system can monitor the load and add more servers and processors as needed. If we see that the average time from collection to reporting is more than a few hours, the system simply adds more servers.

The next stage of the beta testing will be to begin increasing the incoming data next week so we can see how it is working. We expect to send out about 25 invites next week and 50 the week after. If you have not filled out our survey, please do now. Two of the important questions are the number of page views per month your site receives, and what web technology is used for development. Don’t worry if you have very little traffic. We are adding both large and small sites to the mix. Customer feedback is every bit as important as load testing.

I look forward to adding more testers and rolling out the best mobile web analytics product. Thanks for all your help and feedback!

Whether or not you are interested in beta testing, please do fill out our survey. It has help us build a better product, and we will be releasing the summary results after our official launch out of beta.

Click Here for our Survey. It only takes a minute.