Both solutions provide a much needed insight into actual user behavior. PerlMonks is a site that recognizes that ‘Content is King’ but how do the users interact with it? How do people find content on PerlMonks? Considering the feedback I have received to drop IE support, how many people are using IE? Android? and which versions? How long do users stay on PerlMonks? What is the average page load time? Are some pages slower than the rest of the site?

I'd question just how useful the information provided by GA would be to a site like this. Yes, GA can tell you what browsers people are using, but if you're designing for particular browsers, you're already Doing It Wrong. That makes sense for some sites, but I think the overwhelming consensus here would be to design for the standards and assume people will use browsers that meet those standards. "Page load time," if it's useful at all, is better gotten other ways. GA can tell you how long the browser takes to get and render the page, but you can find that out for yourself by, you know, browsing the site. To break page load time down to determine whether slowness is caused by network latency, database lookups, etc., you'd have to run something at the server end. If knowing how people find the site (referrals and search keywords) is useful, that can be gotten from the server logs. Ditto "heatmap testing," as far as what menu links people use; the simple menu design means if you want to know how many people click on "Meditations" (for example), just count the hits on it in the logs.

As for using some open-source alternative: we're all programmers here, right? If you want to know what browsers people are using, why not ask the gods if they'll tell you what LogFormat is being used (assuming Apache), and if they would run a simple script you provided to mine that info from the log for a sample period? The same thing could be done to see what menu items are/aren't used, etc.

For the record, I use GA on most of my client sites (they like to look at the graphs), and sometimes it's useful. I also like a lot of "jQuery crap" and think most concerns about Javascript are a decade out of date. I don't worry much about the privacy issue with GA (although my concern in that area is growing), and I doubt it would add any noticeable load time or browser load to the pages. But I think it's questionable how much it helps you to "know your audience," or how useful it is to do so. With my clients, I find that stats like "bounce rate" are frequently confusing and misleading, and tend to cause people to think the stats are more meaningful than they really are.

To break page load time down to determine whether slowness is caused by network latency, database lookups, etc., you'd have to run something at the server end.

Its been said before (tye?) its the database

If you save-page-as and serve the same page as static page via network, you can see how much faster it loads

Ditto "heatmap testing," as far as what menu links people use; the simple menu design means if you want to know how many people click on "Meditations" (for example), just count the hits on it in the logs.

IIRC the top 3 are The Monastery Gates, Newest Nodes, and Recently Active Threads

The next three are probably "FullPage Chat" (Other CB Clients) and "preview page"

A simple solution would be to either add google analytics to the site and get the data processed for us or run some open source software over the web server logs.

That is not a solution, apache/mod_perl/PerlMonks already keeps logs/stats -- all that would do is share them with google

I'm not sure how hard it would be, but all you have to do is persuade gods to share some awstats generated pages with you

I vaguely recall a long time ago maybe some user-agent strings were posted, but I'm probably misremembering, its not listed on Stats Page

See also Most popular links on homenodes

See also Most popular links on homenodes

How is that helpful? Those are basically bookmarks. And the most-bookmarked pages are not necessarily the most visited pages. Far from it.

(Not to mention that that data is nearly 7 years old.)

How is that helpful?

Its interesting and links to interesting things, that is all.

That is not a solution, apache/mod_perl/PerlMonks already keeps logs/stats -- all that would do is share them with google.

Yes we would be sharing data with google and all the gods would have to do is add a js snippet to the site templates. A small and fast change, easy. Furthermore, google analytics (GA) would give us access to all kinds of search related and end user related data that is not collected by web server software. So yes using GA is a valid solution to this kind of data problem. If your real concern is privacy that is a valid point, which is why I suggested an open source option that can be run locally as well.

I'm not sure how hard it would be, but all you have to do is persuade gods to share some awstats generated pages with you

The purpose of this post was to start a dialog with the gods, pmdev and the community about what stats can do for PM. I know there are plenty of other front and backend developers on PM who use web stats to make better things or are interested in the issue.

Oh, you're back?

Yes we would be sharing data with google and all the gods would have to do is add a js snippet to the site templates. A small and fast change, easy.

Too easy.

Furthermore, google analytics (GA) would give us access to all kinds of search related and end user related data that is not collected by web server software.

But do we need it? PerlMonks seems to get along just fine without it, ever since 1999.
Google knows enough about me already, anyways.

If your real concern is privacy

It is.

that is a valid point,

You'd better believe it.

which is why I suggested an open source option that can be run locally as well.

I think that's already being done, when needed. But why would we need it? The site is mostly compatible with any modern browser, so adding cruft to the templates will only irk people with increased page loading time. By the way, I use Ghostery to block any tracker I see, and I'm sure others do similar things as well.

It's unneeded, I don't want it, and I hope the gods exercise good judgement when opting not to add any unnecessary cruft to the templates.

~Thomas~
confess( "I offer no guarantees on my code." );

Stats are already available, they simply aren't shared with me and you, and as a reader I don't need to see stats

As a pmdev I don't think I need to see stats either. The way I see it, stats are only of any use to gods, and well, gods can see them already --

Are some pages slower than the rest of the site?

yup, and tye has commented on this publicly from time to time ( about tweaking database servers , node cache yada yada )

Regarding GA, read more about such tracking ideas in Facebook 'like' button.

GA needs Javascript. Browsers that can't execute Javascript and browsers that aren't allowed to execute Javascript will not execute that code. So GA does not get all information. How reliable can the output of GA be, given that lack of information? "Garbage in, garbage out".

Apart from that, I once invested some time to dig through parts of the GA Javascript code. I don't remember all details, but it seems that the code sends back a lot of information to Google, and not all of that information seemed to be related to web statistics. It seemed that Google used the GA code to compare what was sent to a real browser with what was send to Google bots.

AWStats is written in Perl. That's about all the positive facts I know about AWStats. When I used it last a few years ago, it made tons of wrong assumptions about sessions and users, and its only real use was to find broken links. But that could also be done with a combination of grep 404 error_log, sed or awk to find the page and referrer urls, and sort -u.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

I believe your request for stats was because you were using fancy CSS for sizing/positioning and wanted to know if the number of IE users was small enough that you could decide to just break the site for them and not care. This puzzled me because it was in the context of you writing a "mobile" version of the site. I didn't think "old IE that sucks at CSS" was an option on a "mobile" device. But I am closer to the opposite of an expert on such matters.

how many people are using ... Android? and which versions?

If I have to cater to not just "Do $this specifically for Android devices" but to a granularity of different versions of Android devices... Well, I won't continue doing that type of work for very long. Seems a very painful way to get stuff done, to me. And I haven't had much problem getting stuff done without wasting my time on such stats.

Considering the feedback I have received to drop IE support, how many people are using IE?

My feedback is not "drop IE support". My feedback is to stop doing things that require a bunch of fancy work-arounds in order for them to work in IE. I don't particularly care if only 1% of our visitors are using IE of old enough vintage to cause problems with using CSS for sizing / positioning, if some of those 1% are long-time users (that perhaps like to use PerlMonks at work on a system where they have very little affect on what browser they can use). Unless the stats said "Nobody has used IE to access PerlMonks in the last 6 months", then the stats aren't going to make me feel comfortable breaking the site for IE. So I wasn't particularly worried about jumping through the hoops to get recent user agent string stats.

How long do users stay on PerlMonks?

The marketing department has been pestering me about that for months.

Are some pages slower than the rest of the site?

Duh. But I don't use browser-based load times for optimizing site performance. It's not really a highly effective way to approach it here. Oh, I've worked with people who built sites that use tons of javascript and load lots of ads from lots of different ad servers and some of those people even created some pretty great tools for getting fine-grained and accurate measurements of what was taking up time loading the web page. And they had a lot of hard work identifying ad servers that were only sometimes really slowing down the page, etc. Nearly none of that applies to PerlMonks and the tiny bits that do are nearly out of my control anyway.

Also doing some short term heatmap testing would help a great deal too.

Marketing is also very concerned about our conversion rate. ;) I'd be more interested in heatmap testing after some actual competent design was applied to site navigation and implemented. Getting stats for optimizing a dysfunctional navigation system would help you tweak it to be slightly more effective of a dysfunctional navigation system.

I believe your request for stats was because you were using fancy CSS for sizing/positioning and wanted to know if the number of IE users was small enough that you could decide to just break the site for them and not care.

That is incorrect. A response to PM redesign: status update suggested I "stop wasting time on IE" and "If the IE numbers are low (and I sure hope they are), a link to google chrome frame is all you need". In the post you are replying to I said the following "Considering the feedback I have received to drop IE support, how many people are using IE? Android? and which versions?" I asked because I want to know and understand the audience better not because I am planning on dropping support for something.

"This puzzled me because it was in the context of you writing a "mobile" version of the site. I didn't think "old IE that sucks at CSS" was an option on a "mobile" device.

I am making a Responsive design for PM which uses media queries to resize elements to better fit the screen size of the browsing device. It is not about old IE sucks at CSS it is about how old IE breaks HTML5 which leads to CSS not getting rendered. I talked about these issues in PM redesign: status update and gave a follow up response to your (tye) question on using CSS hacks to fix IE that goes into more depth about the problem.

My feedback is to stop doing things that require a bunch of fancy work-arounds in order for them to work in IE.

I am using standard, validator compliant markup to build this design. IE is broken period. IE does not follow the standards and that is just the reality of it. IE 9.x still does all kinds of stupid things. Here is a very recent breakdown of how much market share IE has and how bad its standards support in Old browsers are holding back the web. Coding to the standards and adding hacks to get IE working is the best practice for dealing with this kind of problem. Numerous projects like html5shiv, Modernizr, accessifyhtml5, Respond, normalize.css and many others ease the pain of IE and to a much lesser extent other browsers as well. Then we have projects like HTML5 Please, CSS3, please!, caniuse, and litmus which go further in helping chart and solve problems caused by different browsers not implementing the standards. There is simply too much broken in IE to create a modern site without some hacks. More progress is being made.

The marketing department has been pestering me about that for months.

From wikipedia with my emphasis added: Marketing is "the activity, set of institutions, and processes for creating, communicating, delivering, and exchanging offerings that have value for customers, clients, partners, and society at large." Marketing does not mean putting ads on PM. Marketing PM which helps society is not evil or bad.

I'd be more interested in heatmap testing after some actual competent design was applied to site navigation and implemented. Getting stats for optimizing a dysfunctional navigation system would help you tweak it to be slightly more effective of a dysfunctional navigation system.

Yes heatmaps tell us information about the navigation system but they do so much more. All clicks, mouse movements, scrolling, and time information are tracked. What if thousands of users keep clicking on a section of a page they think is a link and it turns out not to be? That is a usability problem. That area can be turned into a link to the expected content, it could be styled different to not look like a link or the general link style can be changed to be less ambiguous. This happens all the time in website design and speculation is not a good replacement for actual hard data from end users.

I agree the current navigation scheme on PM could be a great deal better.

IE is broken period.

IE works just fine here already. Period.

What if thousands of users keep clicking on a section of a page they think is a link and it turns out not to be?

You must be thinking about those sites that take away the one consistent part of Web user interfaces (to replace it with something of their own imagining): Links are underlined. That's not a mistake we made here. Well, except for the two top-line buttons that some people went to quite a lot of effort to eliminate every single visual clue that they actually are buttons. I think that was a mistake. But the lack of visual clues does make it look very nice and uniform (and misleadingly non-functional).

Coding to the standards and adding hacks to get IE working is the best practice for dealing with this kind of problem.

Yes, I'm aware of lots of things that are widely practiced in "modern" web design. Unfortunately, a lot of the reason I am aware of a lot of them is that I have to deal with how f'ed up they tend to make things. Well, not at PerlMonks, thankfully.

I've repeatedly solicited actual good examples of CSS for sizing/positioning. The closest I've ever come was one example that really sucked on one of my browsers to which the submitter asserted that it was due to bugs in my browser. Yeah, cutting edge stuff is often buggy. That doesn't mean that "doesn't work" no longer matters.

So maybe you'll be the first to show me a CSS solution that actually works.

But, no, I'm not convinced that "widely accepted best practice" actually means "good idea" when I constantly see it failing without even trying to look. And when I try to look, it was always easy to find failure.

I consider myself outside the asylum of modern web design. If you actually come up with something that actually works very well, then I'll be overjoyed but very surprised. But all I care about is how well it works and at what cost.

Coding to the standards [....]More progress is being made.

There are plenty of standards to choose from. When you are done with "progress" enough that your newer standards actually "work", then I think it would be a fine time to adopt them. When it comes to choosing standards, I much prefer "it works" over "it is new". Especially when "doesn't work on" includes some of the most prevalent browsers. New standards are nice and can be important in setting direction, etc. And that includes early adopters finding and working out kinks. PerlMonks is not in the business of early adopting new web standards and working out kinks and worrying about market share of versions of browsers.

I'm sure that annoys several people. But it is also appreciated by quite a few people. And it is actually working pretty well, IMO.

- tye        

IE is broken period

All browsers are broken in one way or the other, if you want 100% standard compliance.

As for IE, Except for some small Javascript+DOM issues, IE8 and higher works. And you do not need to support older versions. The oldest Windows versions still supported by Microsoft are XP and Windows Server 2003, both of which can (and should, from a security standpoint) run IE8.

Where you run into trouble with various browsers is fancy CSS positioning and transitioning stuff, table'd layouts on the other hand work just fine and adapt very good to the browser window size.

So, yes, IE has a lot of bugs. But guess what, Firefox has it's own share of rendering bugs and sometimes needs a few plugins to keep it happy. Chrome&Chromium have their own share of problems. Most touch-screen optimized browsers have broken Drag&Drop API's or require special Javascript treatment. And there are the text-only browsers, often used in conjunction with screenreaders or Braille terminals, that may or may not support CSS at all.

So, name me one browser that isn't broken.

Sorry for any bad spelling, broken formatting and missing code examples. During a slight disagreement with my bicycle (which i lost), i broke my left forearm near the elbow. I'm doing the best i can here...