database performance: test setup: Principles
Origins - "The root of all evil"
The idea of running this benchmark came from running the database unit tests of the framework:
noticing a number of discrepancies in the time it took to run the unit tests against individual databases.
Some of those were due to misconfigurations, others were solved by improving the framework and solving locking issues,
but in the end there were still some interresting differences.
What is compared here - "Apples and Oranges"
The setups are as standard as can be.
The only principle guiding the installation of all the sofware is simplicity.
No optimizations, no tweaks, no editing of configuration files.
Individual aspects are compared one at at time (product, kernel, JDK, etc...).
What is not compared here - "KISS - Keep It Simple Stupid"
In short: anything that is not strictly necessary.
Because of the simplicity of the test, this does mean that many related issues will not be covered in this report,
including security/SSL, replication, triggers, transaction isolation, java compilation options, cpu specific optimizations, etc.
The tables and queries used are not "real".
General Methodology Overview - "Shrodinger's Cat"
Preventing the sampling process from having any impact on the data being measured.
The samples are written out after each test is finished,
the amount of data collected in memory is marginal compared to the size of the data handled by the tests.
We ensure that the system load settles before starting a new test and each test run starts with a clean new instance.
Disclaimer - "Shooting the Messenger"
We do not have any commercial links to any of the companies whose products are evaluated here
and have done our best to ensure that the conditions of these tests are as fair as can be,
however no guarantees are given that the results are either accurate or repeatable.
If you decide to use those tests to make a design decision, we strongly recommend
that you run the tests yourself after tuning them to match your expected workload.
All products and services marks contained herein are trademarks or registered trademarks of their respective owners.
Copyright - "Copyleft"
In line with our Open-Source policy, this paper is released without placing restrictions on its re-distribution.
We only ask that credits are given where credits are due.
