It sounds like you have two issues. The first is that you don't want to wait until you have a complete application before benchmarking as you suspect you already know where your bottlenecks will be. The second is you can't think of a good way to synthesize the input without having a full blown application.

I am not sure the approach you are going in is the right one. If I have understood correctly, your ultimate dilema is making an up front design decision about your architecture. It is hard to see why the normal process won't work for you. Perhaps you need to give a more concrete explanation. Regarding your idea about tying your benchmarking to your test suite, it sounds like you want Mock::App (non-existent) where the majority of your API is just stubs but the routines that you are interested in testing and benchmarking are complete. I really don't see what this buys you since a good benchmark should really be comparing the smallest possible variations of functionally equivalent routines.

Update: I see your response and will reply with links and further advice when time permits. For starters, you can take a look at the planning and design phases of the software development life cycle (SDLC).

++

This is exactly the kind of answer I wanted!

I don't actually know what the normal process is. I've just been flying by the seat of my pants for the last decade...

Can you point me to some good urls that I could read so I could find out?

In specific response, how do I profile something that doesn't exist yet? You're exactly right, I'm making an up front design decision. How do I do that?

pileofrogs,
In specific response, how do I profile something that doesn't exist yet? You're exactly right, I'm making an up front design decision. How do I do that?

I have already provided you a link to SDLC and ELISHEVA has begun to expand on phases that come earlier in the process - before writing code, such as plan, design, and specification. All of those things feed into a decision for architecture by fleshing out your requirements. In plain English, what you need to know is that you aren't prepared to start writing code without the real possibility of painting yourself into a corner, failing or perhaps creating a monster.

So without trying to learn the entire SDLC in a day, what you need to do is specify your requirements. Your prototype was too slow - what is fast enough? You said "This means I'm going to have a lot more data and I'm going to search it in a lot more ways" - do you know all the ways you might want to search that data? Do you need to support things like concurrent users (how many) and maintain integrity (ACID transactions)?

What I am getting at is once you know what requirements your solution must meet, you may not have so many choices. Also, unless you have a clear idea of what your performance requirements are, you should base your decision on things you do have specific requirements for (reliability, scalability, maintainability, flexibility, existing expertise, etc).

Finally, if you do come up with specific performance requirements and still can't figure out how to test/benchmark them, come back and see us - that's what we are here for.

Cheers - L~R

Before even thinking about starting any large project I think long and hard about how it can be broken down into smaller testable components. This involves three kinds of analysis:

• data analysis: identify each bit of data you will need. Then figure out how one bit of data relates to another. What "things" are you keeping data about? How are they identified? What attributes do you need to track about these things? Answering these questions will help you break down your project into smaller pieces that can be tested individually. You may find it helpful to dust off whatever you once learned about data normalization. Normalization gives you a particularly robust way of answering these kinds of questions
• use case analysis:Now that you know what information you are working with, how do you need to use it? Is there a work flow involved? If so, what are the steps? If some of your data is derived how will you calcualte it?
• functional analysis: Now that you know both the data and how you want to use it, what are the programming steps for each derivation or work flow component? Do some of the derivations or work flow components have common logic? Can this logic be factored out into smaller reusable subroutines?

These three kinds of analysis are often done in sequence and there are huge arguments in the design field about whether use cases or data come first. However, my feeling is that each kind of analysis simply represents a different way of focusing on the problem and actually answers questions about use, data, and functions all at once. So take the implied sequence with a big grain of salt. When you analyze use cases, you often find that some data that seemed important wasn't and other data that is really important was completely missed. When you analyze data, you often find that there is a lot of data that is very important to getting a job done but its presence is taken as a given so no one notices it is there until they do data analysis. The same thing can happen when you analyze functions - you may discover that an algorithm needs more data than you expected or that a use case that once seemed cut and dry is really a lot more ill defined than anyone realized.

Instead of testing your entire system at once, you can begin by defining objects (and maybe even database tables or flat files) that correspond to each of the "things" you identified during data analysis. Then you can define small tests to make sure that each object and/or data base table is correctly defined. That is, the data you use to construct it is in fact the data that is stored. Then if your actual application creates or modifies multiple objects at once, you can move onto testing entire transactions for creating or modifying objects and database tables (or whatever other form of persistence you choose to use). If one of your anticipated bottlenecks is the creation and storage of data, you can also test this.

In the same way you can test your use cases bit by bit. First you start with writing and testing each of these building block routines. If you expect a particular algorithm to be a bottle neck you can write and then compare the performance of each alternative algorithm. But I'd be a bit careful of your assumptions - sometimes the bottlenecks we expect turn out not to be problems at all. There is a significant risk of premature optimization at this point. My own rule of thumb is don't worry about what is best unless the choice of implementation is going to force some hard-to-back-away from choices when you are ready to combine the building blocks into more complex functionality.

Then when you are comfortable with the behavior of each building block, you can move onto writing and testing larger, more complex functionality. Finally, when you have all of the stages of a calculation or work flow tested, you can move onto testing each use case. This is a good time to apply the strategy suggested by Limbic~Region - profile for bottlenecks, then benchmark. Again, sometimes the bottlenecks you expect aren't the ones that cause the real trouble.

Best of luck with your project, beth

beth,
Finally, when you have all of the stages of a calculation or work flow tested, you can move onto testing each use case. This is a good time to apply the strategy suggested by...

Very astute. I incorrectly assumed the planning and design analysis was a known entity since there was a prototype written several years ago. I have updated my initial response with a link to SDLC and planned to add a more in depth response later if someone hadn't already. Then I saw your response. In reading it, I think there is still additional information that may be valuable to pileofrogs. For large projects needing hardware, COTS licenses, architecture decisions before software development can begin, testing as you have described may not be option. Acquisitions is the part you need to get right the first time because of the cost (time and money) of being wrong. How then can you determine what to buy if it costs money to test which is best?

I know that this is probably very much not the case here as pileofrogs will find that disk I/O is the largest bottleneck and the best way to resolve the issue is to be smart with his database design, caching, and SQL. Assuming a free database and no solid state disks in the budget to eliminate IO, this all falls back to software and your points are spot on. I just think that, by their own admission, being unware of standard process for design decisions should mean doing more research. For instance, learning how to find and interpret independent market research instead of believing what a sales rep tells you about their product being perfect. I don't currently have the time to do this myself but since you seem to have far more experience than myself, if you have links handy that would be great :-)

Cheers - L~R

1. The best indication of how well the program will perform will be how well it's designed (optimise the design and algorithms before the code)
2. Profiling is very useful once the program is written. There are plenty of profilers for perl. I quite like Devel::SmallProf
3. There's the Benchmark module, and another way is to use Timer::HiRes to roll your own. Yet another way would be to set up a $SIG{"ALARM"} handler to set a variable called $finish_loop or whatever. Then call alarm to send the signal some number of seconds in the future, then run your tests in a until ($finish_loop) { # process some number of records, counting how many } loop. This can then give you a measure of the throughput of the code (ie, how many records it can handle per 10 seconds or per second, or whatever).

It seems that all of these really miss the point that your main concern is with finding a suitable storage mechanism. Using a full-blown database using the DBD module might be the best way to go if you're going to need complex queries and updates. If you're not familiar with SQL, you might want to try something like DBM::Deep. It's surprisingly easy to use if you're familiar with using Perl's hashes. The only two things to watch out for are locking files during startup, and writing more complex records/structures into/out of a table, which is handled fairly easily with YAML.

I think that you'll find that the most efficient solution here won't necessarily be the fastest in terms of how many records can be processed per second. Rather, it'll probably come down to how efficiently you can code up the program from requirements and how much time you have to spend debugging and maintaining it. That's really one great advantage of the SQL modules. You might have to put up a fair bit of up-front effort in learning to use it, but once you do, you can become very efficient at dealing with all sorts of data-intensive tasks like the one you've described. In fact, once you've written a few programs that use DBD, you'll probably never want to go back to working with flat files again except to populate the database and generate reports.

Also, while we're on the subject of profiling/benchmarks, remember the old maxim: "Premature optimisation is the root of all evil."