How stats are collected?

[gristgal]

This just occurred to me the other day, and now I can't stop wondering:

If I look at some of my pages using the main page (as a visitor/reader would), will that artificially inflate my statistics? I just added a component yesterday, and I had to look at a sample of pages to ensure it was looking right on all of them (post pages as well as topic pages).

I looked at the hourly stats today, and I don't see a big lump of requests (haven't pulled the server log yet, but I didn't think to check, and I didn't think yesterday to check what my IP addy was; sometimes it changes), but I decided I'd ask you anyway.

Thanks
_________________
"Real knowledge is to know the extent of one's ignorance."
Confucius

[john]

Yes, your Site Stats do include your own visits to the blog. We do not have a provision for you tell the stats generator who you are and not to count you. We do provide the complete server logs as you can perform your own analysis on the data.

[gristgal]

Thanks, I've been downloading the server logs for over a month now. Since I turn off my PC when I'm not doing anything on it, my ISP apparently may reassign my IP#, as I've noted it does change sometimes. What I have to do now is check what my IP is each day, and note it for the following day's log, huh? That way I can subtract out any of my own views.

That's a pain, but it's not worth running up my electric bill in order to keep the same IP#. :sigh:
_________________
"Real knowledge is to know the extent of one's ignorance."
Confucius

[gristgal]

When I went just now to download my stats logs, I noticed that no new stats have been posted since the 15th (Tuesday). You usually have them up the next day. Is it just my blog? If not, when can we expect new ones posted, please?

Thanks

[john]

Our stats processor is a little behind as a result of our system upgrade on Saturday evening and a software error caused by the addition of our caching servers. We've fixed the software error and the stats processor is at work churning through the log files. We should be all caught up shortly.

[gristgal]

Thanks again. I'm afraid I've become fairly dependent on seeing the previous day's activity. Does any other blogging software offer this? I've been thinking it probably didn't. I've had the impression that some other features Blogware offers that are clearly 2nd generation, or maybe 3rd? Like multiple categories, frex, and that some are unique (for the present, at least). It would be irrational to expect all these bells & whistles to come without some cost, whether in money or in patience.

If I'm right about these things, maybe you should emphasize them a bit more heavily?

[gristgal]

I see that I finally got stats for Tuesday & Wednesday, but no server logs! Are they a separate process? And - please forgive my impatience - but when do you expect to be caught up?

I also note that there seems to be something wrong with the server info. The amount shown when I began this online session (before the stats update was processed) plus the two days' usage since posted is not the total shown on the stats page.

[john]

I see that I finally got stats for Tuesday & Wednesday, but no server logs! Are they a separate process?

Yes, they are. I'll check to make sure our developers are looking into this as well.

And - please forgive my impatience - but when do you expect to be caught up?

Your impatience is understandable; for most bloggers stats are, well, addictive... We're expecting to be caught up overnight if all goes according to plan..

I also note that there seems to be something wrong with the server info. The amount shown when I began this online session (before the stats update was processed) plus the two days' usage since posted is not the total shown on the stats page.

I think what you are seeing is that some previous dates had to be revised, since they only included data from the main front end servers and not the new caching servers.

[gristgal]

server logs a separate process:

[john]

Our system is trying to catch up on processing Site Stats, and our admins are trying to push it along. Unfortunately, we are still not caught up to the point where you are able to look at yesterday's stats. Processing stats requires a massive amount of CPU time, in fact even more than required to serve your blogs, and we're doing our best to get back on track.

As for the Web Server logs, yes we're having a problem as admitted yesterday. Downloadable Web Server logs are a brand new feature, having only been introduced in our System Update 3/11/2006. When we introduced our caching servers last week to improve the system performance (and it's been a significant improvement, all blogs are loading much faster today!) and we've had some troubles integrating the log processing from the new caching servers with the main front end servers.

We're very sorry for the delay in solving this, but we've learned the hard way not to push too fast to resolve a problem: it's too easy to create additional problems. So unfortunately we need to be more stringent in testing our fix than we were in testing the feature, hence the delay. We are working to get this resolved as soon as possible, and apologize for the delay.

[gristgal]

You didn't directly address this, so I'm asking: Do I understand that you will not be processing the server logs until you get the stats caught up?

If that is correct, and seeing you've fallen back again a day on mine (none added last night; back to two days behind), I don't think I'm being unduly pessimistic to ask: when you do get to the server logs, will you go back and do them all, or should I anticipate losing some days? Or will you give us more than 7 days for a brief period so we can get them? I would guesstimate they take up a fairly significant amount of space; if you tried to offer more than 7 days' worth on an ongoing basis, you'd be looking at additional costs for storage very shortly. I'm getting vivid flashbacks to the early 80s, when I managed a university department's mainframe usage and storage accounts. :owwww:

If it's possible and feasible (i.e., doesn't require writing a custom sort for the db, and add coding time without reducing processing time significantly) to do updates on some other basis than a "batch job" (to use a now ancient phrase/concept), you might consider asking users to let you know whether they use the logs or not, and just skip over those who don't.

As I mentioned earlier, I didn't start using them until early April. I'm saving them now for analysis, more for purposes of establishing whether/how much continuity of readership I have. And that's why I am so concerned about the point last week - which was the most recent of several changes I've made, with more anticipated - and if/how much change it made/makes in my readership community. If I can't know if a change is happening, I may feel it necessary to put off the other changes until you get this straightened out. It's not as though I intend to do it on an ongoing basis. My intent was to collect the data on a daily basis until a certain period after I stabilize the changes, then spot check weekly - or even perhaps only monthly.

There is, after all, a great deal of information available in the stats readout; the logs are raw data that probably shouldn't be of daily interest indefinitely to anyone who doesn't understand how to use them. Poring obsessively over daily logs over the long term will only make a person crazy. I don't need to be crazy(ier); I just need to know the effect, individual and cumulative, of the changes. It's clear I picked a bad time to make them.

[john]

You didn't directly address this, so I'm asking: Do I understand that you will not be processing the server logs until you get the stats caught up?

Sorry if I was unclear. The two issues are unrelated: the solution to one does not rely on the resolution of the other.

As for Site Stats processing, we're getting closer to being caught up, sorry for the delay. All blogs have been processed through 5/19 and many blogs are caught up to 5/20 (which is where we should be). Looks like your blog is completely caught up.

Also the web server log processing has been restarted, you'll see we've added some additional days to your server logs. We trying to get caught up on these logs with the least inconvenience possible to you and all of our users.

[gristgal]

I'm glad to see the progress. Now I can at least review that part. Obviously, there are still some problems, as stats for the 21st aren't up this morning.

More worryingly, when I looked at server logs, I discovered that the 19th & 20th were there (thanks!), but the 16th, 17th and 18th aren't. Will those still be done, or are they now lost?

Sorry to be such a pain, but I had the misfortune of not being aware that problems were brewing with your computers when I started making changes. Had I had even the least clue that logs might not be available, I'd have delayed both of the changes. The 16th was the first day of both, and - with a continued topic - logs for the 16th through the 18th are essential for me to know whether there was continuity, and what possible changes - if any - occurred.

[john]

Our admins have assured me that no web server logs will be lost, that we are still working on the next days worth of Site Stats, and that everything will be resolved shortly. Thanks for your patience.

[gristgal]

Thank you. I really, really hate bugging you like this. I assure you I wouldn't if the timing of it weren't so crucial. I will try to wait patiently.