Frequently Asked Questions (FAQs)

Who we are?

What is the SPEC Research Group?

The Research Group is a technical group within SPEC established to serve as a platform for collaborative research efforts in the area of quantitative system evaluation and analysis.

What is the mission of the Research Group?

The mission of the Research Group is to promote innovative research in the area of quantitative system evaluation and analysis by serving as a platform for collaborative research efforts fostering the interaction between industry and academia in the field. The scope of the group includes computer benchmarking, performance evaluation, and experimental system analysis in general, considering both classical performance metrics such as response time, throughput, scalability and efficiency, as well as other non-functional system properties included under the term dependability, for example, availability, reliability, and security. The conducted research efforts span the design of metrics for system evaluation as well as the development of methodologies, techniques and tools for measurement, load testing, profiling, workload characterization, dependability and efficiency evaluation of computing systems. For further information, see Research Group Mission Statement.

How can I join the Research Group?

If you are interested in joining, you can learn about membership here.

Meetings

When does the Research Group meet?

The Research Group has a monthly General Meeting (virtual); in addition, each RG Working Group has its own separate (typically biweekly) virtual meetings. Representatives of the member organizations also meet face-to-face at conferences and workshops sponsored or organized by the Research Group. A regular annual meeting is normally organized co-located with the ACM/SPEC International Conference on Performance Engineering (ICPE).

How are members of the Research Group Steering Committee selected?

The Research Group Steering Committee (RGSC) is elected by active members once a year. At most two individuals from the same member organization may be elected to the RGSC in a given year. Here is the current list of elected members of the RGSC.

How can I participate as a guest or observer before joining the Research Group?

You can contact the Chair or one of the other officials of the Research Group Steering Committee (RGSC) listed here. Here is the current membership of the RGSC.

Do I need to be a RG Member to participate in virtual meetings?

If you would like to attend as a Guest before you join, please email SPEC staff with your interest and they will connect you to the appropriate Working Group Chairs or a member of the Research Group Steering Committee (RGSC). You will be required to complete a Guest Participation form before you can join a face-to-face meeting or virtual.

If my company is a member of the SPEC Research Group, can I participate in the virtual meetings and how do I obtain dial-in information?

You should contact your company’s SPEC representative to obtain the information that you need to participate in the virtual meetings. If you already have access to the restricted-access SPEC wiki pages, the information to access the virtual meeting is provided there.

Is there a minimum contribution (in terms of hour/month) from each member?

The Charter of the Research Group does not specify a number of hours that a member organization is required to contribute to the Research Group activities. The level of participation may vary from an observer providing feedback on important decisions, to an active contributor to a given project, to project leader and/or working group chair.

The participation of each organization in the regular meetings is recorded. In order to have voting status in a meeting, an organization must had been represented in at least two consecutive meetings immediately prior to meeting in which the vote takes place.

Research Working Groups

What is a Working Group?

A Working Group focuses on the development of research benchmarks or tools in a specific area of technology.

What are the current working groups?

Here is a list of current Research Working Groups.

What is the process to start a new Research Working Group? Can anyone submit proposals or only members of the Research Group?

The first step is to submit a proposal to the Research Group Steering Committee (RGSC). It is advisable to consult with the Research Group Chair before preparing a proposal. Here is the list of current officers of the Research Group. Normally, proposals are submitted by member organizations, but occasionally the RGSC may invite a guest participant to present a proposal.

Are there Committees in addition to Working Groups?

In other SPEC groups technical committees are established to develop and support it’s new benchmark suites. At the moment, the Research Group has not established any committees.

Scope of Activities

What types of activities is the Research Group involved in?

The scope of the RG includes computer benchmarking, performance evaluation, and experimental system analysis considering both classical performance metrics such as response time, throughput, scalability and efficiency, as well as other non-functional system properties included under the term dependability, e.g., availability, reliability, and security. The conducted research efforts span the design of metrics for system evaluation as well as the development of methodologies, techniques and tools for measurement, load testing, profiling, workload characterization, dependability and efficiency evaluation of computing systems.

Other ongoing activities of the RG include publishing a newsletter, maintaining a repository of peer-reviewed tools for quantitative system evaluation and analysis, establishing a portal for benchmarking-related topics, recognizing outstanding research contributions (e.g., SPEC Kaivalya Dixit Distinguished Dissertation Award), and organizing conferences and workshops.

Does the Research Group publish papers?

The Research Group sponsors the ACM/SPEC International Conference on Performance Engineering (ICPE). The papers accepted to this conference are published in the ACM Digital Library as well as on the SPEC RG website. Additionally, research articles and white papers relevant to the activities of the Research Group are linked from the publications area of the Research Group Website.

Benchmarks and Peer-Reviewed Tools

How can I obtain a benchmark suite or a peer-reviewed tool published by the Research Group?

Benchmarks and peer-reviewed tools published by the Research Group will be available in the Tools section.

What is the difference between conventional SPEC benchmarks and benchmarks published by the Research Group?

The most important difference is that benchmarks published by the Research Group are not intended to serve as benchmarks for direct comparison and marketing of existing products. The goal is rather to provide representative application scenarios, defined at a higher level of abstraction, that can be used as a basis to evaluate early prototypes and research results as well as full-blown implementations in the respective technology domain. Research benchmarks can be defined both for existing technologies as well as for new technologies at the early stages of their inception before full-fledged industrial implementations are available.

Do I need to be a member to download benchmarks or tools published by the Research Group?

The research benchmarks and tools published by the Research Group are public. You do not need to be a member to download them.

Can benchmarks published by the Research Group be used for comparison and marketing of products?

No. The main intent of the benchmarks and tools published by the Research Group is to foster research. Once an area becomes mature, the creation of a benchmark suite to be used for direct comparison and marketing of products will follow the standard SPEC process for creation of benchmarks.

What is a peer-reviewed tool?

The RG maintains a repository of peer-reviewed tools for quantitative system evaluation and analysis. The published tools undergo a thorough review process by multiple independent experts to ensure high quality and relevance to the community. The review process covers important quality factors, including maturity, availability and usability. The scope of the considered tools covers system evaluation and analysis with respect to both classical performance metrics, such as response time, throughput, scalability and efficiency, as well as other non-functional system properties included under the term dependability, e.g., availability, reliability, and security. In particular, tools for measurement, monitoring, profiling, workload characterization, load testing, stress testing and resilience testing are solicited. Of particular interest are generic tools that address recurring issues in benchmark development as well as reusable software components that significantly reduce the effort for building new benchmarks. The focus is on open-source domain-independent tools that have been well tested and shown to provide generic functionality that can be reused in future benchmark development. In addition, benchmark development frameworks targeted at specific application domains are considered. Any organization or individual is eligible to submit a proposal for publishing a tool in the repository.

What infrastructure does the Research Group offer to host a benchmark or peer-reviewed tool?

The Research Group maintains a permanent repository for research benchmarks and peer-reviewed tools.

What are the benefits to publish a tool or benchmark suite via the Research Group?

There are many advantages to the publication of a tool or a benchmark suite with the Research Group. The reviewing process for publication is likely to provide the author with valuable feedback about the tool or benchmark. Once it is accepted, the submission will be kept in a repository maintained by the Research Group. The visibility and acceptance of the benchmark or tool by the community is likely to benefit significantly from its publication by the Research Group.

Can a tool be published via the Research Group as well as in other repositories such as sourceforge?

Yes. However, the Research Group will only maintain the version published on the SPEC server.

What is the process to update a tool after its initial publication?

Project updates are classified into minor or major updates. Minor updates deliver bug fixes and smaller new features. Major updates include new functionality, significant new features or a project redesign. A version-control repository is maintained and both types of updates must be marked by tags in this repository. The maintainer is responsible to verify that the update conforms to the RG rules and that all necessary parts are ready for public download after each published update. Each update must be explicitly listed in the project’s downloadable/versioned documentation by a list of fixed bugs and added features. The most recent project update (date and version) must be posted on the project’s site.

Benchmark and Tool Submission and Publication Process

How can I propose a benchmark suite to the Research Group?

The first step is to present a proposal to the Research Group Steering Committee. Please contact the RGSC Chair prior to preparing such a proposal. Here is the list of current officers of the Research Group.

What is the process for publishing a tool?

Any organization or individual is eligible to submit a proposal for publishing a tool on the Research Group’s Website. Submitted proposals undergo a review process by the Research Group Steering Committee (RGSC) prior to publication. After an initial discussion by the RGSC, the proposal will be assigned a group of reviewers to rigorously review the proposal and the tool. The reports from the reviewers are then used by the RGSC to make a decision about publishing the tool. The tool submission and publication process are described in detail here.

Is membership in the Research Group a prerequisite or can anyone submit tool proposals?

Membership is not required for the submission of a tool proposal.

How are decisions about acceptance/rejection of proposals for benchmarks or tools communicated?

Upon request of the proponent, the tool submission will be regarded as SPEC confidential material and will be only accessible to members of the Research Group Steering Committee (RGSC) and designated reviewers. The decision of acceptance/rejection will be confidentially communicated by the RGSC Chair to the proponent.

Proposals can be submitted per email to the Research Group Release Manager. See the current officers of the Research Group.

Miscellaneous

The Research Group sponsors workshops and conferences in the area of computer benchmarking, performance evaluation and quantitative system evaluation and analysis, in general. Currently, the most prominent event sponsored by the Research Group is the ACM/SPEC International Conference on Performance Engineering.