The best option would seem to be logging the number of user each second, but your concerned with the size of the datastructure this would create.

Does it really have to be so big?

Assuming that you have a reliable way of determining when a user session ends if they just disconnect without logging off--which is a problem you'd have to solve with any solution--if your maximum user count at any given time is less than 65k, storing a 16-bit count in a tied array for each second of the day is going to take 168k/day. The nice thing is that retrieval would be very fast.

If that's too much, then you could store 16-bit count at the top of each days file showing the number of sessions when the 'day' started and then store an 8-bit/second (or even a 4-bit/second) delta giving a filesize of 84k (42k)depending on how far back you need to keep on-line, this shouldn't be too much. Again the retreival would be fast, although not as fast as you'd need to process the days file from the beginning. With 8-bit delta you would allow for a maximum change of logins of ±128 users in any given second which is probably ample for most sites. If the delta grew beyond this value in any given second, you could write the maximum for the delta chosen in that given second and carry any remainder over to the next, that allows your running totals to remain correct at the cost of minor inaccuracies during periods of heavy usage.

You might even take this a step further and store 7 days (or how ever far back you need to go) in a single file and then rotate through it ensuring that you only ever need a set size datastore updating the total count each time you start to overwrite the deltas. The logic gets a little tricky if you need to get data for periods in the file prior to the last update to the master count, as you would then need to read backwards and invert the signs on the deltas.

The downside of this is that you would have to(potentially) read through through an entire rotation periods data to calculate the current number. However, given say 7 days worth of 8-bit deltas, a quick experiment show that perl can slurp the 600k file in a couple of seconds and apply the deltas to the base count in 11 seconds.

If 16-bits is enough to hold your absolute number of sessions, and you can afford the 168k/day filesize, the retrieval of any given second would be very nearly instantaneous. You could always archive the files to tape or CD if you need to. Even from a CD, the calculation for any second, or even a whole hour would be trivial and pretty fast.