|
Re: Graph a file without using arrays
by kyle (Abbot) on Sep 19, 2008 at 19:53 UTC
|
If you're worried about some array using too much memory, I recommend using DBM::Deep. With it, you can tie an array to a file on disk so that its contents are there instead of in memory. Hand the array over to GD::Graph, and it shouldn't know the difference (but it may gobble a lot of memory for its own purposes).
You could try progressively larger data sets to see how memory usage expands with the input data. That might give you an idea of whether the whole data set will be too much.
| [reply] |
Re: Graph a file without using arrays
by Perlbotics (Archbishop) on Sep 19, 2008 at 21:27 UTC
|
Hi, do you really need to process each individual sample from your input file?
Will each sample be visually distinguishable in the output graph?
E.g. cramming 15000 samples into an inch of a 300 dpi plot is overkill.
Depending on your demands, you might be able to reduce the input
by averaging some samples before plotting, e.g. by means of a sliding window?
| [reply] |
Re: Graph a file without using arrays
by zentara (Cardinal) on Sep 19, 2008 at 20:23 UTC
|
You could use a Tk::Canvas and read from a file, making a rectangle or line segment for every read, see Tk Realtime data aquisition It reads from a socket, but it would be easy to switch to a file read. If you like the thing, you can save the canvas to postscript, then convert to any format you would want.
| [reply] |
Re: Graph a file without using arrays
by CountZero (Bishop) on Sep 19, 2008 at 20:30 UTC
|
The file contains so much data How much is so much? Did you actually try it?
CountZero A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James
| [reply] |
|
|
Yes, I did. The system started to slow down. The amount of data will depend on the size of the file(s) and how many files are chosen.
| [reply] |
|
|
Thanks, but it still doesn't answer my question "How much data?" A few thousand, million, ...? It is interesting to find out where the practical limits of Perl are and how to solve such problems.
CountZero A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James
| [reply] |
Re: Graph a file without using arrays
by GrandFather (Saint) on Sep 19, 2008 at 21:52 UTC
|
As Perlbotics implies, there is a fundamental disconnect between what you want to do and the way you are trying to do it. It simply does not make sense to try and display a vast amount of data on a small canvas. Perhaps you should tell us more about the nature of the data and how you want to view it so we can suggest better ways of handling the problem.
If you are plotting lots of lines then the following technique may work for you: divide the horizontal span into some number of columns (close to the number of x pixels you plan to plot). For each column generate a histogram count (with close to y pixels number of bins) of the number of lines that pass through each bin. Plot an intensity value for each bin that is a function of the bin count.
Perl reduces RSI - it saves typing
| [reply] |
Re: Graph a file without using arrays
by BrowserUk (Patriarch) on Sep 20, 2008 at 09:30 UTC
|
It is pretty easy to create a graph yourself using GDs drawing primitives, as you read the file, though you would need to make two passes through the file. The first to calculate minimums & maximums so you know how to scale the data.
But still the problem remains. If there are so many values that loading them into ram is a problem, then plotting them all on a single graph will either result in graph so dense it is unreadable, or so large that the image size itself could be a memory problem.
Without you saying
- What type of graph.
- What the data looks like.
- How many values.
There's really not a lot anyone can do to help.
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply] |
|
|
In response to Grandfather and BrowserUK…
I have files that contain ping travel times between nodes. I want to be able to graph this data using scatter graphs. (I would also like to create histograms with logarithmic scales. I have not found out how to do this in perl, so any advice you can give would be appreciated) The files contain over 8600 lines of data. Each line contains a timestamp and the time too and from the test site. I am not concerned about being able to see each single point. Most of the data will be grouped together and will form a line. This is just to be able to see a visual distribution of travel time in relation to time of day.
| [reply] |
|
|
The files contain over 8600 lines of data.
8640 data points (I'm assuming that you are pinging once every 10 seconds over each 24 hr period?), is not a lot of data. There really is no need to do you own thing. I just generated a 8640 timestamps/random times and plotted them on a 1024x768 points graph, and the total memory usage was well under 10 MB for the program, data and graph.
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply] |
|
|
|
|
|