 |
|
 |
 |
|
Procedures for Testing Computer Performance
Objectives
The main objective of testing PC performance is to obtain objective (test) data on how fast computers are in various applications, so that the computers tested could be compared. Thus, splitting the general procedure into several narrower modifications allows us to skip performance comparisons in those tasks, where such computers are not intended to be used by the manufacturer itself. As a results, such comparisons, even if done, make absolutely no sense.
Results
The result of testing PC performance according to one of the procedures is an average performance score, obtained by averaging the results of tests in a given procedure. So, the only correct comparison is between scores (of computers), obtained by the same test procedure.
Test Procedures
Applications
3D Modeling and Rendering
- 3ds max 7.0
- Maya 6.5
- Lightwave 8.5 x64 Edition
This class of applications traditionally belongs to resource-intensive software mostly used by professionals. 3D modeling packages in render mode heavily load a processor, in interactive mode they load much a CPU-RAM system and 3D accelerator. Practically all 3D modeling packages require much memory (more than on an average desktop computer). As a rule, they are not critical to HDD performance. They can use all processors in a system.
Archivers
- WinRAR 3.51
- 7-Zip 4.32 x64 Edition
These programs are used practically by everybody, irregardless of their professional or amateur status. They are critical to CPU/memory speed and also depend on HDD performance. They are indifferent to a video card. Most archivers do not support multiprocessing yet, but our test procedure includes one of the exceptions (7-Zip).
Audio Encoders
- LAME 3.98
- Monkey Audio 4.01
- OGG Encoder 2.8
- Windows Media Encoder 9 x64 Edition
A vast class of programs for compressing audio files, lossy or lossless. They are mostly used by home users. As a rule, this class of programs is critical to CPU performance and architecture; the influence of HDD, memory, and video card is minimal. Multiprocessing is rarely supported (but Windows Media Encoder offers this feature).
CAD/CAM/CAE
- MATLAB 7.1
- Pro/ENGINEER Wildfire 2.0
- SolidWorks 2005
These programs are used practically solely by professionals, as their learning curve is very steep and they are very expensive. They are intended to be used in manufacturing for modeling various processes or for designing various units (as a replacement to Whatman paper, Kuhlman drafting unit, and pencil). They are critical to performance of all PC components without exception, they are very exigent to RAM size.
Compilers
- Microsoft Visual C++ Professional 6.0
One of the wide-spread tasks is compiling programs, written in C++. Microsoft compiler is a standard de facto for programmers, who write programs for Microsoft Windows OS. The load is evenly distributed between a processor, memory, and HDD. The hard drive is defragmented before the tests to lessen the effect of the latter. As a matter of fact, many home users also write programs from time to time. But on the whole, it's a professional test.
Synthetic and semi synthetic tests
- CPU RightMark 2005 Lite x64 Edition
- RightMark Memory Analyzer
These programs mostly evaluate maximum PC performance in relatively simple frequent operations. Results of these tests are not directly related to PC performance in real applications. But they can be used for express comparison of computers (with certain reservations, of course.) CPU RightMark mostly tests CPU performance, RightMark Memory Analyzer—memory performance.
PC games
- F.E.A.R. 1.3
- Half-Life 2
- Unreal Tournament 2004 build 3339
- Quake 4 Point Release 1.1
It's a popular class of applications for home computers. It would have been unwise to exclude it from our test procedure, despite its "frivolous" nature. As a rule, games are more critical to performance of processors and video cards. Memory performance is less important. HDD performance has most often no direct effect on gaming. It affects only loading game levels (that's why HDD effect is not taken into account in this test procedure).
OCR
- FineReader Professional 8.0
This task is mostly typical of home and office computers. It's critical to CPU performance, contribution of the other computer systems into the overall OCR speed is much less critical.
Bitmap editors
- Adobe Photoshop CS2 (9.0)
As this class of programs is represented in our test procedure by Adobe Photoshop, we can speak of professional nature of this test, because home and office users usually edit photos in simple programs. This test is critical to CPU and RAM performance, it can use more than one processor (if available). A very important role is also played by memory size. It must be significantly larger than in an average home or office computer.
Video encoders
- Canopus ProCoder 2.01.30
- VirtualDubMod 1.5.1.1
- DivX 6.1.1
- Windows Media Video 9 VCM
- x264 v.438
- XviD 1.1.0 Release
A popular task for a home computer. However, some of these programs are also used professionally. Video encoders are generally critical to CPU architecture and performance, to a lesser degree—memory. Some of them can use more than one processor.
Web server
- Apache 2.0.55 for Windows
Purely professional software (even if used for amateur objectives, it's still professional, as it requires a high level of expertise in its configuration). It copes well with a lot of processors. It's critical to CPU and memory performance (in various proportions depending on a type of server load). Theoretically, this test can be critical to HDD performance, but our test procedure does not take it into account.
Integrated benchmarks, based on real applications
One of the most popular all-in-one benchmarks. In fact, we try to avoid third-party tests in our procedures. But we made an exception for this package, as its office score is highly adequate and its tests cover lots of popular office software.
Types of the procedure
Abridged procedure
- Archivers
- Synthetic and semi synthetic tests
- PC games
- Integrated benchmarks, based on real applications
A minimum set of tests that evaluate approximate performance of a computer in a short period of time and get the results as soon as possible. If a computer is stable and adequate, the abridged test procedure can be completed for 2-3 hours (here and below we mean the time for tests without taking into account assemblage and configuration).
Procedure for home and office computers
- Archivers
- Audio encoders
- Synthetic and semi synthetic tests
- PC games
- OCR
- Video encoders
- Integrated benchmarks, based on real applications
They adequately evaluate PC performance from the point of view of home and/or office usage. Test duration depending on PC performance—from 6 to 8 hours.
Test procedure for professional computers and workstations
- Archivers
- 3D Modeling and Rendering
- CAD/CAM/CAE
- Compilers
- Synthetic and semi synthetic tests
- Bitmap editors
- Video encoders
- Web server
It allows to evaluate performance of a powerful desktop computer or a workstation from the point of view of suitability for professional usage "in the most general case". Professionals may be interested in the results of separate tests, so in our opinion, publishing separate results as well as the average score is absolutely justified. The tests will take up to 1.5 working days.
Full (maximum details) procedure
It includes all the above tests. Maximum of useful information, but quite a long period of testing (about two-three working days).
Test time estimates
All time estimates are given for the worst case scenario with some "safety margin" for minor issues. A unit of time equals 10 minutes.
But of course, it holds true only when a testbed works more or less fine, without serious problems.
- 3ds max 7.0 — 3 hours
- Maya 6.5 — 3 hours
- Lightwave 8.5 x64 Edition — 1.5 hours
- WinRAR 3.51 — 10 minutes
- 7-Zip 4.32 x64 Edition — 15 minutes
- LAME 3.98 — 10 minutes
- Monkey Audio 4.01 — 20 minutes
- OGG Encoder 2.8 — 10 minutes
- Windows Media Encoder 9 x64 Edition — 10 minutes
- MATLAB 7.1 — 10 minutes
- Pro/ENGINEER Wildfire 2.0 — 3 hours
- SolidWorks 2005 — 3 hours
- Microsoft Visual C++ Professional 6.0 — 20 minutes
- CPU RightMark 2005 Lite x64 Edition — 10 minutes
- F.E.A.R. 1.3 — 10 minutes
- Half-Life 2 — 10 minutes
- Unreal Tournament 2004 build 3339 — 10 minutes
- Quake 4 Point Release 1.1 — 10 minutes
- FineReader Professional 8.0 — 30 minutes
- Adobe Photoshop CS2 (9.0) — 1.5 hours
- Canopus ProCoder 2.01.30 — 20 minutes
- DivX 6.1.1 — 1 hour
- Windows Media Video 9 VCM — 1.5 hours
- x264 v.438 — 1 hour
- XviD 1.1.0 Release — 1.5 hours
- Apache 2.0.55 for Windows — 10 minutes
- SYSmark 2004SE — 2 hours
- RightMark Memory Analyzer — 10 minutes
PC assemblage and configuration (hardware only) — 1 hour
Restoring a test image, installing additional drivers, getting the system ready for running the tests (additional configuration, defragmentation) — 1 hour
Standard testing hours
- 3D Modeling and Rendering — 4.0
- Archivers — 0.5
- Audio Encoders — 1.0
- CAD/CAM/CAE — 3.0
- Compilers — 0.5
- Synthetic and semi synthetic tests — 0.5
- PC games — 0.5
- OCR — 0.5
- Bitmap editors — 1.0
- Video encoders — 2.0
- Web server — 0.5
- Integrated benchmarks, based on real applications — 1.0
- Assembling a computer, checking and configuring BIOS settings — 1.0
- Restoring an image, installing drivers, configuring OS — 1.0
An attempt to solve problems on one's own (without consulting a technical director) and subsequent writing of a bug report — NO MORE THAN 1.0 per system
Here is the point:
if there is a real problem, a test engineer can try to cope with it on his own, without resorting to the help of a technical director — but we shall not pay for a whole working day of fumbling with a system, if he reports about a problem only post factum.
If he fails to find a solution for an hour, he must call a technical director.
|
|
|
|
|
