Thursday, 27 July 2017

Table of Contents

Frequently Asked Questions (FAQs)

  • Who we are?

      • What is the SPEC Research Group?

    The Research Group is a new technical group within SPEC established to serve as a platform for collaborative research efforts in the area of quantitative system evaluation and analysis.

      • What is the mission of the Research Group?

    The mission of the Research Group is to promote innovative research in the area of quantitative system evaluation and analysis by serving as a platform for collaborative research efforts fostering the interaction between industry and academia in the field. The scope of the group includes computer benchmarking, performance evaluation, and experimental system analysis in general, considering both classical performance metrics such as response time, throughput, scalability and efficiency, as well as other non-functional system properties included under the term dependability, e.g., availability, reliability, and security. The conducted research efforts span the design of metrics for system evaluation as well as the development of methodologies, techniques and tools for measurement, load testing, profiling, workload characterization, dependability and efficiency evaluation of computing systems. For further information, see Research Group Mission Statement.

  • Membership and Meetings:

      • How can I join the Research Group?

    Any company or organization interested in the work of the Research Group  is welcome to join. You must pay the membership fees, be willing to abide by SPEC's bylaws, the Fair Use Rules, the Guidelines for Handling SPEC Information, and RG's Charter.

    If you would like to join, please complete the membership application form and fax it to 1-703-579-8463 or email it to info(at)spec(dot)org.

    The application will be submitted for approval by the SPEC Board, which takes approximately 14 days. When the approval process is completed, you will be notified of your application's status and the next steps.

      • What kind of organizations are members of the Research Group?

    Current members in the Research Group include major corporations in the area of electronics and information technology, major research institutes and major universities from North America, Europe, Asia, and Australia. Here is a list of current Research Group member organizations

      • Is it possible to join the Research Group as an individual rather than as an organization?

    Currently membership in the Research Group is only possible at the organization level. Once an organization joins the RG, any member of the organization can participate in the SPEC RG activities.

      • Are there different categories of membership to the Research Group?

    All member organizations have the same membership category.

      • When does the Research Group meet?

    The Research Group meets bi-weekly via teleconference. Representatives of the member organizations also meet face-to-face at conferences and workshops sponsored or organized by the Research Group. A regular annual meeting is normally organized co-located with the ACM/SPEC International Conference on Performance Engineering (ICPE).

      • How are members of the Research Group Steering Committee selected?

    The Research Group Steering Committee (RGSC) is elected by active members once a year. At most two individuals from the same member organization may be elected to the RGSC in a given year. Here is the current list of elected members of the RGSC.

      • Do I need to be a member to participate in conference calls?

    Guests are allowed to participate in the conference calls as long as they sign a Guest Participation Statement to be filed with the SPEC head office.

      • How can I participate as a guest or observer before joining the Research Group?

    You should contact the chair or one of the other officials of the Research Group Steering Committee (RGSC). Here is the current membership of the RGSC.

      • If my company is a member of the SPEC Research Group, can I participate in conference calls and how do I obtain dial-in information?

    You should contact your company's SPEC representative to obtain the information that you need to participate in the conference calls. If you already have access to the restricted-access SPEC wiki pages, the information to access the teleconference is provided there.

      • How much does it cost to join the Research Group?

    The annual fee to join is $200 US Dollars.

      • Is there a minimum contribution (in terms of hour/month) from each member?

    The Charter of the Research Group does not specify a number of hours that a member organization is required to contribute to the Research Group activities. The level of participation may vary from an observer providing feedback on important decisions, to an active contributor to a given project, to project leader and/or working group chair.

    The participation of each organization in the regular meetings is recorded. In order to have voting status in a meeting, an organization must had been represented in at least two consecutive meetings immediately prior to meeting in which the vote takes place.

  • Research Working Groups:

      • What is a Working Group?

    A Working Group focuses on the development of research benchmarks or tools in a specific area of technology.

      • What are the current working groups?

    Here is a list of current Research Working Groups.

      • What is the process to start a new Research Working Group? Can anyone submit proposals or only members of the Research Group?

    The first step is to submit a proposal to the Research Group Steering Committee (RGSC). It is advisable to consult with the Research Group Chair before preparing a proposal. Here is the list of current officers of the Research Group. Normally, proposals are submitted by member organizations, but occasionally the RGSC may invite a guest participant to present a proposal.

      • Are there sub-committees in addition to working groups?

    In other SPEC groups sub-committees are established as a more permanent organization that looks after a benchmark that has been published. At the moment the Research Group has not established any sub-committees.

  • Scope of Activities:

      • What types of activities is the Research Group involved in?

    The scope of the RG includes computer benchmarking, performance evaluation, and experimental system analysis considering both classical performance metrics such as response time, throughput, scalability and efficiency, as well as other non-functional system properties included under the term dependability, e.g., availability, reliability, and security. The conducted research efforts span the design of metrics for system evaluation as well as the development of methodologies, techniques and tools for measurement, load testing, profiling, workload characterization, dependability and efficiency evaluation of computing systems.

    A major component of RG is the development of standard scenarios and workloads—called research benchmarks—for emerging technologies and applications. Benchmarks from the RG are intended primarily for in-depth quantitative analysis and evaluation of early prototypes and research results. This differentiates them from conventional benchmarks used for direct comparison and marketing of existing products.

    Other ongoing and planned activities of the RG include publishing a newsletter and journal, establishing a portal for benchmarking-related resources, recognizing outstanding research contributions, and organizing conferences and workshops.

      • Does the Research Group publish papers?

    The Research Group sponsors the ACM/SPEC International Conference on Performance Engineering (ICPE). The papers accepted to this conference are published in the ACM Digital Library as well as on the SPEC RG website. Additionally, research articles and white papers relevant to the activities of the Research Group are linked from the publications area of the Research Group Website.

  • Benchmarks and Peer-Reviewed Tools:

      • How can I obtain a benchmark suite or a peer-reviewed tool published by the Research Group?

    Benchmarks and peer-reviewed tools published by the Research Group will be available in the Tools section of the Research Group Website.

      • What is the difference between conventional SPEC benchmarks and benchmarks published by the Research Group?

    The most important difference is that benchmarks published by the Research Group are not intended to serve as benchmarks for direct comparison and marketing of existing products. The goal is rather to provide representative application scenarios, defined at a higher level of abstraction, that can be used as a basis to evaluate early prototypes and research results as well as full-blown implementations in the respective technology domain. Research benchmarks can be defined both for existing technologies as well as for new technologies at the early stages of their inception before full-fledged industrial implementations are available.

      • Do I need to be a member to download benchmarks or tools published by the Research Group?

    The research benchmarks and tools published by the Research Group are public. You do not need to be a member to download them.

      • Can benchmarks published by the Research Group be used for comparison and marketing of products?

    No. The main intent of the benchmarks and tools published by the Research Group is to foster research. Once an area becomes mature, the creation of a benchmark suite to be used for direct comparison and marketing of products will follow the standard SPEC process for creation of benchmarks.

      • What is a peer-reviewed tool?

    The RG maintains a repository of peer-reviewed tools for quantitative system evaluation and analysis. The published tools undergo a thorough review process by multiple independent experts to ensure high quality and relevance to the community. The review process covers important quality factors, including maturity, availability and usability. The scope of the considered tools covers system evaluation and analysis with respect to both classical performance metrics, such as response time, throughput, scalability and efficiency, as well as other non-functional system properties included under the term dependability, e.g., availability, reliability, and security. In particular, tools for measurement, monitoring, profiling, workload characterization, load testing, stress testing and resilience testing are solicited. Of particular interest are generic tools that address recurring issues in benchmark development as well as reusable software components that significantly reduce the effort for building new benchmarks. The focus is on open source domain-independent tools that have been well tested and shown to provide generic functionality that can be reused in future benchmark development. In addition, benchmark development frameworks targeted at specific application domains are considered. Any organization or individual is eligible to submit a proposal for publishing a tool in the repository.

      • What infrastructure does the Research Group offer to host a benchmark or peer-reviewed tool?

    The Research Group maintains a permanent repository for research benchmarks and peer-reviewed tools.

      • What are the benefits to publish a tool or benchmark suite via the Research Group?

    There are many advantages to the publication of a tool or a benchmark suite with the Research Group. The reviewing process for publication is likely to provide the author with valuable feedback about the tool or benchmark. Once it is accepted, the submission will be kept in a repository maintained by the Research Group. The visibility and acceptance of the benchmark or tool by the community is likely to benefit significantly from its publication by the Research Group.

      • Can a tool be published via the Research Group as well as in other repositories such as sourceforge?

    Yes. However, the Research Group will only maintain the version published on the SPEC server.

      • What is the process to update a tool after its initial publication?

    Project updates are classified into minor or major updates. Minor updates deliver bug fixes and smaller new features. Major updates include new functionality, significant new features or a project redesign. A version-control repository is maintained and both types of updates must be marked by tags in this repository. The maintainer is reponsible to verify that the update conforms to the RG rules and that all necessary parts are ready for public download after each published update. Each update must be explicitly listed in the project's downloadable/versioned documentation by a list of fixed bugs and added features. The most recent project update (date and version) must be posted on the project's site.

  • Benchmark and Tool Submission and Publication Process:

      • How can I propose a benchmark suite to the Research Group?

    The first step is to present a proposal to the Research Group Steering Committee. Please contact the RGSC Chair prior to preparing such a proposal. Here is the list of current officers of the Research Group.

      • What is the process for publishing a tool?

    Any organization or individual is eligible to submit a proposal for publishing a tool on the Research Group's Website. Submitted proposals undergo a review process by the Research Group Steering Committee (RGSC) prior to publication. After an initial discussion by the RGSC, the proposal will be assigned a group of reviewers to rigorously review the proposal and the tool. The reports from the reviewers are then used by the RGSC to make a decision about publishing the tool. The tool submission and publication process is described in detail on the following page.

      • Is membership in the Research Group a prerequisite or can anyone submit tool proposals?

    Membership is not required for the submission of a tool proposal.

      • How are decisions about acceptance/rejection of proposals for benchmarks or tools communicated?

    Upon request of the proponent, the tool submission will be regarded as SPEC confidential material and will be only accessible to members of the Research Group Steering Committee (RGSC) and designated reviewers. The decision of acceptance/rejection will be confidentially communicated by the RGSC Chair to the proponent.

      • How can I submit a paper to be listed on the SPEC Research Group’s website?

    Proposals can be submitted per email to the Research Group Release Manager. See the current officers of the Research Group.

  • Miscellaneous:

      • Does the Research Group sponsor/endorse conferences?

    The Research Group sponsors workshops and conferences in the area of computer benchmarking, performance evaluation and quantitative system evaluation and analysis, in general. Currently the most prominent events sponsored by the Research Group are the ACM/SPEC International Conference on Performance Engineering and the IEEE International Conference on Autonomic Computing.