Cray XE6m (Copper)
User Guide

Table of Contents

1. Introduction

1.1. Document Scope and Assumptions

This document provides an overview and introduction to the use of the Cray XE6m Open Research System, Copper, located at the ERDC DSRC, along with a description of the specific computing environment on Copper. The intent of this guide is to provide information that will enable the average user to perform computational tasks on the system. To receive the most benefit from the information provided here, you should be proficient in the following areas:

  • Use of the UNIX operating system
  • Use of an editor (e.g., vi or emacs)
  • Remote usage of computer systems via network or modem access
  • A selected programming language and its related tools and libraries

1.2. Policies to Review

Users are expected to be aware of the following policies for working on Copper.

1.2.1. Login Node Abuse Policy

The login nodes, copper01-copper02, provide login access for Copper and support such activities as compiling, editing, and general interactive use by all users. Consequently, memory or CPU-intensive programs running on the login nodes can significantly affect all users of the system. Therefore, only small applications requiring less than 10 minutes of runtime and less than 8 GBytes of memory are allowed on the login nodes. Any job running on the login nodes that exceeds these limits may be unilaterally terminated.

1.2.2. Workspace Purge Policy

Close management of space in the /work file system is a high priority. Files in the /work file system that have not been accessed in 30 days are subject to the purge cycle. If available space becomes critically low, a manual purge may be run, and all files in /work are eligible for deletion. Using the touch command (or similar commands) to prevent files from being purged is prohibited. Users are expected to keep up with file archival and removal within the normal purge cycles.

Note! If it is determined as part of the normal purge cycle that files in your $WORKDIR directory must be deleted, you WILL NOT be notified prior to deletion. You are responsible to monitor your workspace to prevent data loss.

1.3. Obtaining an Account

The process of getting an account on the HPC systems at any of the DSRCs begins with getting an account on the HPCMP Portal to the Information Environment, commonly called a "pIE User Account." If you do not yet have a pIE User Account, please visit HPC Centers: Obtaining An Account and follow the instructions there. If you need assistance with any part of this process, please contact the HPC Help Desk at

1.4. Requesting Assistance

The HPC Help Desk is available to help users with unclassified problems, issues, or questions. Analysts are on duty 7:00 a.m. - 7:00 p.m. Central, Monday - Friday (excluding Federal holidays).

You can contact the ERDC DSRC directly in any of the following ways for after-hours support.

  • E-mail:
  • Phone: 1-800-500-4722 or (601) 634-4400
  • Fax: (601) 634-5126
  • U.S. Mail:
    U.S. Army Engineer Research and Development Center
    ATTN: CEERD-IH-D HPC Service Center
    3909 Halls Ferry Road
    Vicksburg, MS 39180-6199

For more detailed contact information, please see our Contact Page.

2. System Configuration

2.1. System Summary

Copper is a Cray XE6m. The login and compute nodes are populated with AMD Opteron processors. Copper uses a dedicated Cray Gemini high-speed network for MPI messages and IO traffic. Copper uses Lustre to manage its parallel file system that targets arrays of SAS disk drives. Copper has 460 compute nodes that share memory only on the node; memory is not shared across the nodes. Each compute node has two sixteen-core processors (32 cores per node) that operate under a Cray Linux Environment (CLE) sharing 64 GBytes of DDR3 memory, with no user-accessible swap space. Copper is rated at 138 peak TFLOPS and has 239 TBytes (formatted) of parallel disk storage.

Copper is intended to be used as a batch-scheduled HPC system. Its login nodes are not to be used for large computational (e.g., memory, IO, long executions) work. All executions that require large amounts of system resources must be sent to the compute nodes by batch job submission.

Node Configuration
Login Nodes Compute Nodes
Total Nodes 2 460
Operating System SUSE Linux CLE
Cores/Node 16 32
Core Type AMD 64-bit Opteron AMD Interlagos Opteron
Core Speed 2.7 GHz 2.3 GHz
Memory/Node 128 GBytes 64 GBytes
Accessible Memory/Node 8 GBytes 60 GBytes
Memory Model Shared on node. Shared on node.
Distributed across cluster.
Interconnect Type Ethernet Cray Gemini
File Systems on Copper
Path Capacity Type
/usr/local ($HOME)15 TBytesLustre
/work ($WORKDIR)224 TBytesLustre
/archive204 TBytesNFS

2.2. Processors

Copper uses 2.7-GHz AMD model 8384 processors on its login nodes. There are four processors per node, each with four cores, for a total of 16 cores per login node. These quad-core processors have 128 KBytes of L1 cache per core, 512 KBytes of L2 cache per core, and 6144 KBytes of shared L3 cache.

Copper uses 2.3-GHz AMD 6200 series processors on its compute nodes. There are two processors per node, each with sixteen cores, for a total of 32 cores per node. By design the AMD 6200 series processor pairs cores into modules that allow them to share an L1 instruction cache, an L2 cache, and a "Flex FP" FPU (Floating Point Unit). Each module (two cores) has a 64-KByte L1 instruction cache and a 2-MByte L2 cache. All cores on a processor have their own 16-KByte L1 data cache and share a 16-MByte L3 cache.

2.3. Memory

Copper uses both shared- and distributed-memory models. Memory is shared among all the cores on a node, but is not shared among the nodes across the cluster.

Each login node contains 128 GBytes of main memory. All memory and cores on the node are shared among all users who are logged in. Therefore, users should not use more than 8 GBytes of memory at any one time.

Each compute node contains 64 GBytes of memory, of which 60 GBytes is user-accessible shared memory.

2.4. Operating System

The operating system on Copper's login nodes is SUSE Enterprise Linux. The compute nodes use a reduced functionality Linux kernel that is designed for computational computing. The combination of these two operating systems is known as the Cray Linux Environment (CLE). The compute nodes can provide access to dynamically shared objects and most of the typical Linux commands and basic functionality by including the Cluster Compatibility Mode (CCM) option in your PBS batch submission script or command. This increase in capability is included in Cray's CLE-3.0 and higher operating environment. See sections 6.4 and 6.5 for more information on using CCM.

2.5. File Systems

Copper has the following file systems available for user storage:

2.5.1. /usr/local/

This file system is locally mounted from Copper's login and compute nodes. It has a formatted capacity of 15 TBytes. All users have a home directory (30-GByte quota) which may be referenced by the environment variable $HOME.

2.5.2. /work

This file system is locally mounted from Copper's login and compute nodes. It has a formatted capacity of 224 TBytes. All users have a /work directory which may be referenced by the environment variable $WORKDIR.

Raid / Striping Concerns for Large Files

It is important to note that /work is a parallel, striped file system. This means that as files are written, they are automatically divided into chunks and written across multiple disk sets, or "OSTs," simultaneously. This process, called "striping," plays a vital role in running very large jobs because it significantly improves file I/O speed, thereby reducing the time required to read or write a file. Without parallel striping, large jobs, many of which require hundreds of GBytes of disk space, would spend much of their time just reading from and writing to disk.

The default stripe size for files in the /work file system is 1 MByte, and the default stripe count is two stripes. Increasing the stripe count is advisable when creating files that are larger than 40 GBytes, and reducing the stripe count is recommended when writing many small files. The I/O strategy of the application is also a factor in deciding the stripe count. As a rule, when writing a large volume of data, an application should try to use all the OSTs. If writing a single file, set the stripe count to the number of OSTs. When writing more files than there are OSTs, set the stripe count to 1. If the number of files being written is fewer than the number of OSTs, set the stripe count so that all OSTs will be used. For an explanation of how to do this and recommendations for striping specific file sizes, see the document, "The Lustre File System on Copper."

You can also see $SAMPLES_HOME/Data_Management/OST_Stripes on Copper for an example of how to increase stripe counts in a batch job.

2.6. Peak Performance

Copper is rated at 138 peak TFLOPS or 9.2 GFLOPS per core.

3. Accessing the System

3.1. Kerberos

A Kerberos client kit must be installed on your desktop to enable you to get a Kerberos ticket. Kerberos is a network authentication tool that provides secure communication by using secret cryptographic keys. Only users with a valid HPCMP Kerberos authentication can gain access to Copper. More information about installing Kerberos clients on your desktop can be found at HPC Centers: Kerberos & Authentication.

3.2. Logging In

  • Kerberized SSH
    % ssh (copper0[1-2])
  • Kerberized rlogin is also allowed.

3.3. File Transfers

File transfers to DSRC systems (except for those to the local archive server) must be performed using Kerberized versions of the following tools: scp, mpscp, sftp, ftp, and kftp. Before using any Kerberized tool, you must use a Kerberos client to obtain a Kerberos ticket. Information about installing and using a Kerberos client can be found at HPC Centers: Kerberos & Authentication.

The command below uses secure copy (scp) to copy a single local file into a destination directory on a Copper login node. The mpscp command is similar to the scp command, but has a different underlying means of data transfer, and may enable greater transfer rate. The mpscp command has the same syntax as scp.

% scp local_file (# = 1 to 2)

Both scp and mpscp can be used to send multiple files. This command transfers all files with the .txt extension to the same destination directory. More information about mpscp can be found at the Overview of mpscp page on the ERDC DSRC website.

% scp *.txt (# = 1 to 2)

The example below uses the secure file transfer protocol (sftp) to connect to Copper, then uses the sftp cd and put commands to change to the destination directory and copy a local file there. The sftp quit command ends the sftp session. Use the sftp help command to see a list of all sftp commands.

% sftp (# = 1 to 2)

sftp> cd target_dir
sftp> put local_file
sftp> quit

The Kerberized file transfer protocol (kftp) command differs from sftp in that your username is not specified on the command line, but given later when prompted. The kftp command may not be available in all environments.

% kftp (# = 1 to 2)

username> user
kftp> cd target_dir
kftp> put local_file
kftp> quit

Windows users may use a graphical file transfer protocol (ftp) client such as Filezilla.

4. User Environment

4.1. User Directories

The following user directories are provided for all users on Copper.

4.1.1. Home Directory

When you log on to Copper, you will be placed in your home directory, /usr/local/u/username. The environment variable $HOME is automatically set for you and refers to this directory. $HOME is visible to both the login and compute nodes and may be used to store small user files, but it has limited capacity and is not backed up; therefore, it should not be used for long-term storage.

4.1.2. Work Directory

Copper has one large file system (/work) for the temporary storage of data files needed for executing programs. You may access your personal working directory under /work by using the $WORKDIR environment variable, which is set for you upon login. Your $WORKDIR directory has no disk quotas. Because of high usage, the /work file system tends to fill up frequently. Please review the Purge Policy and be mindful of your disk usage.

REMEMBER: /work is a "scratch" file system and is not backed up. You are responsible for managing files in your $WORKDIR by backing up files to the archive server and deleting unneeded files when your jobs end. See the section below on Archive Usage for details.

All of your jobs should execute from your $WORKDIR directory, not $HOME. While not technically forbidden, jobs that are run from $HOME are subject to disk space quotas and have a much greater chance of failing if problems occur with that resource.

To avoid unusual errors that can arise from two jobs using the same scratch directory, a common technique is to create a unique subdirectory for each batch job by including the following lines in your batch script:

mkdir -p ${TMPD}

4.2. Shells

The following shells are available on Copper: csh, bash, ksh, tcsh, zsh, and sh. To change your default shell, please email a request to Your preferred shell will become your default shell on the Copper cluster within 1-2 working days.

4.3. Environment Variables

A number of environment variables are provided by default on all HPCMP HPC systems. We encourage you to use these variables in your scripts where possible. Doing so will help to simplify your scripts and reduce portability issues if you ever need to run those scripts on other systems.

4.3.1. Login Environment Variables

The following environment variables are common to both the login and batch environments:

Common Environment Variables
Variable Description
$ARCHIVE_HOME Your directory on the archive server
$ARCHIVE_HOST The host name of the archive server
$BC_HOST The generic (not node specific) name of the system.
$CC The currently selected C compiler. This variable is automatically updated when a new compiler environment is loaded.
$COST_HOME This variable contains the path to the base directory of the default installation of the Common Open Source Tools (COST) installed on a particular compute platform. (See BC policy FY13-01 for COST details.)
$CSI_HOME The directory containing the following list of heavily used application packages: ABAQUS, Accelrys, ANSYS, CFD++, Cobalt, EnSight, Fluent, GASP, Gaussian, LS-DYNA, MATLAB, and TotalView, formerly known as the Consolidated Software Initiative (CSI) list. Other application software may also be installed here by our staff.
$CXX The currently selected C++ compiler. This variable is automatically updated when a new compiler environment is loaded.
$DAAC_HOME The directory containing DAAC-supported visualization tools: ParaView, VisIt, and EnSight.
$F77 The currently selected Fortran 77 compiler. This variable is automatically updated when a new compiler environment is loaded.
$F90 The currently selected Fortran 90 compiler. This variable is automatically updated when a new compiler environment is loaded.
$HOME Your home directory on the system
$JAVA_HOME The directory containing the default installation of JAVA
$KRB5_HOME The directory containing the Kerberos utilities
$PET_HOME The directory containing the tools formerly installed and maintained by the PET staff. This variable is deprecated and will be removed from the system in the future. Certain tools will be migrated to $COST_HOME, as appropriate.
$PROJECTS_HOME A common directory where group-owned and supported applications and codes may be maintained for use by members of a group. Any project may request a group directory under $PROJECTS_HOME.
$SAMPLES_HOME The Sample Code Repository. This is a collection of sample scripts and codes provided and maintained by our staff to help users learn to write their own scripts. There are a number of ready-to-use scripts for a variety of applications.
$WORKDIR Your work directory on the local temporary file system (i.e., local high-speed disk).
4.3.2. Batch-Only Environment Variables

In addition to the variables listed above, the following variables are automatically set only in your batch environment. That is, your batch scripts will be able to see them when they run. These variables are supplied for your convenience and are intended for use inside your batch scripts.

Batch-Only Environment Variables
Variable Description
$BC_CORES_PER_NODE The number of cores per node for the compute node on which a job is running.
$BC_MEM_PER_NODE The approximate maximum user-accessible memory per node (in integer MBytes) for the compute node on which a job is running.
$BC_MPI_TASKS_ALLOC The number of MPI tasks allocated for a job.
$BC_NODE_ALLOC The number of nodes allocated for a job.

4.4. Modules

Software modules are a convenient way to set needed environment variables and include necessary directories in your path so that commands for particular applications can be found. Copper uses "modules" to initialize your environment with COTS application software, system commands and libraries, compiler suites, environment variables, and PBS batch system commands.

A number of modules are loaded automatically as soon as you log in. To see the modules which are currently loaded, use the "module list" command. To see the entire list of available modules, use "module avail". You can modify the configuration of your environment by loading and unloading modules. For complete information on how to do this, see the Modules User Guide.

4.5. Archive Usage

All of our HPC systems have access to an online archival mass storage system that provides long-term storage for users' files on a petascale tape file system that resides on a robotic tape library system. A 204-TByte disk cache frontends the tape file system and temporarily holds files while they are being transferred to or from tape.

Tape file systems have very slow access times. The tapes must be robotically pulled from the tape library, mounted in one of the limited number of tape drives, and wound into position for file archival or retrieval. For this reason, users should always tar up their small files in a large tarball when archiving a significant number of files. A good maximum target size for tarballs is about 200 GBytes or less. At that size, the time required for file transfer and tape I/O is reasonable. Files larger than 1 TByte will span more than one tape, which will greatly increase the time required for both archival and retrieval.

The environment variable $ARCHIVE_HOME is automatically set for you and can be used to reference your mass storage archive directory when using archive commands. The command getarchome can be used to display the value of $ARCHIVE_HOME.

4.5.1. Archival Command Synopsis

A synopsis of the archive utility is listed below. For information on additional capabilities, see the Archive User Guide or read the online man page that is available on each system. This command is non-Kerberized and can be used in batch submission scripts if desired.

5. Program Development

5.1. Programming Models

Copper supports five parallel programming models: Message Passing Interface (MPI), Shared-MEMory (SHMEM), Open Multi-Processing (OpenMP), and Partitioned Global Address Space (PGAS): Co-Array FORTRAN, and Unified Parallel C (UPC). A Hybrid MPI/OpenMP programming model is also supported. MPI and SHMEM are examples of the message- or data-passing models, while OpenMP uses only shared memory on a node by spawning threads. PGAS programming using Co-Array FORTRAN and Unified Parallel C shares partitioned address space, where variables and arrays can be directly addressed by any processing element.

Cray provides the Cray Application Developer's Environment (CADE) User's Guide as a complete programmer's reference for all supported programming models.

5.1.1. Message Passing Interface (MPI)

This release of MPI-2 derives from Argonne National Laboratory MPICH-2 and implements the MPI-2.2 standard except for spawn support, as documented by the MPI Forum in "MPI: A Message Passing Interface Standard, Version 2.2."

On Cray systems, MPI is a component of the Cray Message Passing Toolkit (MPT), which is a software package that supports parallel programming across a network of computer systems through a technique known as message passing. MPI establishes a practical, portable, efficient, and flexible standard for message passing that makes use of the most attractive features of a number of existing message-passing systems, rather than selecting one of them and adopting it as the standard. See "man intro_mpi" for additional information.

When creating an MPI program on Copper, ensure the following:

To compile an MPI program, use the following examples:

ftn -o mpi_program mpi_program.f         ## for Fortran, or
cc -o mpi_program mpi_program.c          ## for C

The program can then be launched using the aprun command, as follows:

aprun -n mpi_procs mpi_program [user_arguments]

where mpi_procs is the number of MPI processes being started. For example:

#### starts 64 mpi processes; 32 on each node, one per core
## request 2 nodes, each with 32 cores
#PBS -l select=2:ncpus=32:mpiprocs=32
aprun -n 64 ./a.out

Or simply:

aprun -B ./a.out

Accessing More Memory Per MPI Process

By default, one MPI process is started on each core of a node. This means that on Copper, the available memory on the node is split 32 ways. A common concern for MPI users is the need for more memory for each process. To allow an individual process to use more of the node's memory, you need to allow some cores to remain idle.

The example below demonstrates several ways to increase memory per process by modifying PBS directives and aprun command line options. Suppose you were running 64 processes (2 nodes with a full 32 processes per node).

#PBS -l select=2:ncpus=32:mpiprocs=32
aprun -n 64 ./a.out

The example below doubles the memory per node by cutting the total number of processes and processes started per node in half. To change the number of processes started per node, adjust the value of mpiprocs in the PBS directive, and add the "-N" option to the aprun command. To change the total number of processes, adjust the "-n" option.

#PBS -l select=2:ncpus=32:mpiprocs=16
aprun -n 32 -N 16 ./a.out

Instead of "-N 16", use the "-d" option to skip every other core.

aprun -n 32 -d 2 ./a.out

Or use the "-B" option if you'd like to specify the number of processes using PBS directives only, with no need for -n, -d, nor -N.

aprun -B ./a.out

For more information about aprun, see the aprun man page.

5.1.2. SHared MEMory (SHMEM)

The logically shared, distributed-memory access (SHMEM) routines provide high-performance, high-bandwidth communication for use in highly parallelized scalable programs. The SHMEM data-passing library routines are similar to the MPI library routines: they pass data between cooperating parallel processes. The SHMEM data-passing routines can be used in programs that perform computations in separate address spaces and that explicitly pass data to and from different processes in the program.

The SHMEM routines minimize the overhead associated with data-passing requests, maximize bandwidth, and minimize data latency. Data latency is the length of time between a process initiating a transfer of data and that data becoming available for use at its destination.

SHMEM routines support remote data transfer through "put" operations that transfer data to a different process and "get" operations that transfer data from a different process. Other supported operations are work-shared broadcast and reduction, barrier synchronization, and atomic memory updates. An atomic memory operation is an atomic read and update operation, such as a fetch and increment, on a remote or local data object. The value read is guaranteed to be the value of the data object just prior to the update. See "man intro_shmem" for details on the SHMEM library.

When creating a pure SHMEM program on Copper, ensure the following:

  • That the MPICH2 module is not loaded. To check this, run the "module list" command. If cray-mpich is listed, use the following command:

    module unload cray-mpich

  • That the logically shared distributed memory access routines (module cray-shmem) are loaded. To check this, run the "module list" command. If cray-shmem is not listed, use the following command:

    module load cray-shmem

  • That the source code includes one of the following lines:

    INCLUDE 'mpp/shmem.fh'  ## for Fortran, or
    #include <mpp/shmem.h>  ## for C

To compile a SHMEM program, use the following examples:

ftn -o shmem_program shmem_program.f90 ## for Fortran, or
cc -o shmem_program shmem_program.c ## for C

The ftn and cc wrappers resolve all SHMEM routine calls automatically. Specific mention of the SHMEM library is not required on the compilation line.

The program can then be launched using the aprun command as follows:

aprun -n N shmem_program [user_arguments]

where N is the number of processes being started, with each process utilizing one core. The aprun command launches executables across a set of CNL compute nodes. When each member of the parallel application has exited, aprun exits. For more information about aprun, type "man aprun".

5.1.3. Open Multi-Processing (OpenMP)

OpenMP is a portable, scalable model that gives programmers a simple and flexible interface for developing parallel applications. It supports shared-memory multiprocessing programming in C, C++, and Fortran, and consists of a set of compiler directives, library routines, and environment variables that influence compilation and run-time behavior.

When creating an OpenMP program on Copper, ensure the following:

To compile an OpenMP program, use the following examples:

For C codes:

cc -o OpenMP_program -mp=nonuma OpenMP_program.c  ## PGI 
cc -o OpenMP_program -h omp OpenMP_program.c      ## Cray 
cc -o OpenMP_program -openmp OpenMP_program.c     ## Intel
cc -o OpenMP_program -fopenmp OpenMP_program.c    ## GNU

For Fortran codes:

ftn -o OpenMP_program -mp=nonuma OpenMP_program.f ## PGI 
ftn -o OpenMP_program -h omp OpenMP_program.f     ## Cray 
ftn -o OpenMP_program -openmp OpenMP_program.f    ## Intel
ftn -o OpenMP_program -fopenmp OpenMP_program.f   ## GNU

See section 5.2 for additional information on available compilers.

When running OpenMP applications, the $OMP_NUM_THREADS environment variable must be used to specify the number of threads. For example:

aprun -d 32 ./OpenMP_program [user_arguments]

In the example above, the application starts the OpenMP_program on one node and spawns a total of 32 threads. Since Copper has 32 cores per compute node, this yields 1 thread per core.

5.1.4. Hybrid Processing (MPI/OpenMP)

An application built with the hybrid model of parallel programming can run on Copper using both OpenMP and Message Passing Interface (MPI). In hybrid applications, OpenMP threads can be spawned by MPI processes, but MPI calls should not be issued from OpenMP parallel regions or by an OpenMP thread.

When creating a hybrid (MPI/OpenMP) program on Copper, follow the instructions in the MPI and OpenMP sections above for creating your program. Then use the compilation instructions for OpenMP.

Use the aprun command and the $OMP_NUM_THREADS environment variable to run a hybrid program. You may need aprun options "-n", "-N", and "-d" to get the desired combination of MPI processes, nodes, and cores.

aprun -n mpi_procs -N mpi_procs_per_node -d threads_per_mpi_proc mpi_program

Note the product of mpi_procs_per_node and threads_per_mpi_proc (-N * -d) should not exceed 32, the number of cores on a Copper node.

In the following example, we want to run 8 MPI processes, and each MPI process needs about half the memory available on a node. We therefore request 4 nodes X 32 cores. We also want each MPI process to launch 8 OpenMP threads, so we set the environment variable accordingly and assign 8 threads per MPI process in the aprun command.

####  MPI/OpenMP on 4 nodes, 8 MPI processes with 8 threads each
## request 4 nodes, each with 32 cores
#PBS -l select=4:ncpus=32:mpiprocs=2
export OMP_NUM_THREADS=8      ## create 8 threads per MPI process
aprun -n 8 -N 2 -d 8 ./xthi.x ## assigns 8 MPI processes with 2 MPI processes per node

Note, in this example, each thread is assigned to a core, and half the cores will remain idle. Alternatively, if we wanted each MPI process to launch 16 threads, we would do the following:

#PBS -l select=4:ncpus=32:mpiprocs=2
export OMP_NUM_THREADS=16  
aprun -n 8 -N 2 -d 16 ./xthi.x

Or simply:

aprun -B ./xthi.x

In this example, each node gets two MPI processes, and all cores are assigned a thread. See the aprun man page for more detail on how MPI processes and threads are allocated on the nodes.

5.1.5. Partitioned Global Address Space (PGAS)

The Cray Fortran compiler supports Co-Array Fortran (CAF), and the Cray C compiler supports Unified Parallel C (UPC). These are PGAS extensions that enable the user to reference memory locations on any node, without the need for message-passing protocols. This can greatly simplify writing and debugging a parallel code. These compilers also allow the user to combine PGAS programming constructs with the flexibility of message-passing protocols. The PGAS extensions are not available for the PGI, Intel, or GNU compilers.

Cray Fortran and C reference manuals currently refer the reader to external sources for details on the CAF and UPC concepts and syntax. Users should follow the Cray document links in the Links to Vendor Documentation at the end of this guide to locate these reference manuals. The manuals will provide further links to the external sources.

Compilation of UPC and CAF codes is straightforward. Make sure to swap the programming environment module:

module swap PrgEnv-pgi PrgEnv-cray

Then, simply use the standard Cray compilers with the following flags:

ftn -o myprog -h caf  myprog.f    ## for Fortran
cc -o myprog -h upc  myprog.c     ## for C

Use the aprun command to execute the program as described above for MPI programs:

#PBS -l select=2:ncpus=32:mpiprocs=32
aprun -n 64 ./myprog

Or simply:

aprun -B ./myprog

Many users of PGAS extensions will also use MPI or SHMEM calls in their codes. In such cases, be sure to use the appropriate include statements in your source code, as described above.

5.2. Available Compilers

Copper has four programming environment suites.

On Copper, different sets of compilers are used to compile codes for serial vs. parallel execution.

Compiling for the Compute Nodes

Codes compiled to run on the compute nodes may be serial or parallel. The x86-64 instruction set for AMD Interlagos processors has extensions for the Floating Point Unit (FPU) that require the module craype-interlagos to be loaded. This module is loaded for you by default. To compile codes for execution on the compute nodes, the same compile commands are available in all programming environment suites as shown in the following table:

Compute Node Compiler Commands
Language PGI Cray Intel GNU Serial/Parallel
C cc cc cc cc Serial/Parallel
C++ CC CC CC CC Serial/Parallel
Fortran 77 f77 f77 f77 f77 Serial/Parallel
Fortran 90 ftn ftn ftn ftn Serial/Parallel
Compiling for the Login Nodes

Codes may be compiled to run on the login nodes in one of two ways. Either replace the craype-interlagos module with the craype-target-native module and use the compiler commands from the table above, as follows:

module swap craype-interlagos craype-target-native
cc myprog.c .-o myprog.x

Note: If you use the above compiler commands with the craype-interlagos module loaded, your executable will error with "illegal command".

Or, use the serial compiler commands from the table below.

Serial-Only Compiler Commands
Language PGI Cray Intel GNU Serial/Parallel
C pgcc craycc icc gcc Serial
C++ pgCC crayCC icpc g++ Serial
Fortran 77 pgf77 crayftn ifort gfortran Serial
Fortran 90 pgf90 crayftn ifort gfortran Serial
Changing Compiler Suites

The PGI programming environment is loaded for you by default. To use a different suite, you will need to swap modules. See Relevant Modules (below) to learn how.

5.2.1. Portland Group (PGI) Compiler Suite

The PGI Programming Environment provides a large number of options that are the same for all compilers in the suite. The following table lists some of the more common options that you may use:

PGI Compiler Options
-c Generate intermediate object file but do not attempt to link.
-I directory Search in directory for include or module files.
-L directory Search in directory for libraries.
-o outfile Name executable "outfile" rather than the default "a.out".
-Olevel Set the optimization level. For more information on optimization, see the section on Profiling and Optimization.
-Mfree Process Fortran codes using free form.
-i8, -r8 Treat integer and real variables as 64-bit.
-Mbyteswapio Big-endian files; the default is for little-endian.
-g Generate symbolic debug information.
-Mbounds Add array bound checking.
-Minfo=all Reports detailed information about code optimizations to stdout as compile proceeds.
-Mlist Generate a file containing the compiler flags used and a line numbered listing of the source code.
-mp=nonuma Recognize OpenMP directives.
-Bdynamic Compiling using shared objects requires CCM mode for execution on compute nodes.
-Ktrap=* Trap errors such as floating point, overflow, and divide by zero (see man page).
-fPIC Generate position-independent code for shared libraries.

Detailed information about these and other compiler options is available in the PGI compiler (pgf95, pgcc, and pgCC) man pages on Copper.

5.2.2. Cray Compiler Environment

The Cray compiler has a long tradition of high performance compilers for excellent vectorization (it vectorizes more loops than other compilers) and cache optimization (automatic blocking and automatic management of what stays in cache).

The Partitioned address space (PGAS) languages such as Unified Parallel C (UPC) and Co-Array Fortran are supported on Copper via the Cray compiler.

The following table lists some of the more common options that you may use:

Cray Compiler Options
-c Generate intermediate object file but do not attempt to link.
-I directory Search in directory for include or module files.
-L directory Search in directory for libraries.
-o outfile Name executable "outfile" rather than the default "a.out".
-Olevel Set the optimization level. For more information on optimization, see the section on Profiling and Optimization.
-f free Process Fortran codes using free form.
-h byteswapio Big-endian files; the default is for little-endian.
-g Generate symbolic debug information.
-s integer64
-s real64
Treat integer and real variables as 64-bit.
-s default64 Pass -s integer64, -s real64 to compiler.
(set by default) Recognize OpenMP directives (disable "-h noomp").
-h upc ( only C) Recognize UPC.
-h caf Recognize Co-Array Fortran.
-h dynamic Compiling using shared objects requires CCM mode for execution on compute nodes.
-Ktrap=* Trap errors such as floating point, overflow, and divide by zero (see man page).
-fPIC Generate position-independent code for shared libraries.

Detailed information about these and other compiler options is available in the Cray compiler (crayftn, craycc, crayCC) man pages on Copper.

5.2.3. Intel Compiler Environment

The following table lists some of the more common options that you may use:

Intel Compiler Options
-c Generate intermediate object file but do not attempt to link.
-I directory Search in directory for include or module files.
-L directory Search in directory for libraries.
-o outfile Name executable "outfile" rather than the default "a.out".
-Olevel Set the optimization level. For more information on optimization, see the section on Profiling and Optimization.
-free Process Fortran codes using free form.
-convert big_endian Big-endian files; the default is for little-endian.
-g Generate symbolic debug information.
-openmp Recognize OpenMP directives.
-Bdynamic Compiling using shared objects requires CCM mode for execution on compute nodes.
-fpe-all=0 Trap floating point, divide by zero, and overflow exceptions.
-fPIC Generate position-independent code for shared libraries.

Detailed information about these and other compiler options is available in the Intel compiler (ifort, icc, icpc) man pages on Copper.

5.2.4. GNU Compiler Collection

The GNU Programming Environment provides a large number of options that are the same for all compilers in the suite. The following table lists some of the more common options that you may use:

GNU Compiler Options
-c Generate intermediate object file but do not attempt to link.
-I directory Search in directory for include or module files.
-L directory Search in directory for libraries.
-o outfile Name executable "outfile" rather than the default "a.out".
-Olevel Set the optimization level. For more information on optimization, see the section on Profiling and Optimization.
-g Generate symbolic debug information.
-fconvert=big-endian Big-endian files; the default is for little-endian.
Turns on increased error reporting.

Detailed information about these and other compiler options is available in the GNU compiler (gfortran, gcc, and g++) man pages on Copper.

5.3. Relevant Modules

By default, Copper loads the PGI programming environment for you. The Cray CCE, Intel, and GNU environments are also available. To use either of these, the PGI module must be unloaded and replaced with the one you wish to use. To do this, use the "module swap" command as follows:

module swap PrgEnv-pgi PrgEnv-cray          ## To switch to Cray CCE
module swap PrgEnv-pgi PrgEnv-intel         ## To switch to Intel
module swap PrgEnv-pgi PrgEnv-gnu           ## To switch to GNU

In addition to the compiler suites, all of these modules also load the MPICH2 and LibSci modules. The MPT module initializes MPI. The LibSci module includes solvers and single-processor and parallel routines that have been tuned for optimal performance on Cray XE systems (BLAS, LAPACK, ScaLAPACK, etc.). For additional information on the MPT and LibSci modules, see the intro_mpi and intro_libsci man pages on Copper.

The table below shows the naming convention for various programming environment modules.

Programming Environment Modules
Module Module Name
PGI PrgEnv-pgi
Cray CCE PrgEnv-cray
Intel PrgEnv-intel
GNU PrgEnv-gnu

Under each programming environment, the compiler version can be changed. With the default PGI programming environment, for example, the compiler version can be changed from the default to version 10.0.0 with this command:

module swap pgi pgi/10.0.0

Use the "module avail" command to see all the available compiler versions for PGI, Cray CCE, Intel, and GNU.

A number of Cray-optimized libraries (e.g., FFTW, HDF5, NetCDF, and PETSc) are available on Copper with associated module files to set-up the necessary environment. As the environment depends on the active PrgEnv-* module, users should load library-related module files after changing the PrgEnv-*module.

When using SHMEM, load the cray-shmem module, as follows:

module load cray-shmem

For more information on using modules, see the Modules User Guide.

5.4. Libraries

Cray's LibSci and AMD's Core Math Library (ACML) are both available on Copper. In addition, an extensive suite of math and science libraries are available in the $PET_HOME directory.

5.4.1. Cray LibSci

Copper provides Cray's LibSci library as part of the modules that are loaded by default. This library is a collection of single-processor and parallel numerical routines that have been tuned for optimal performance on Cray XE systems. The LibSci library contains optimized versions of many of the BLAS math routines as well as Cray versions of most of the ACML routines. Users can utilize the LibSci routines, instead of the public domain or user written versions, to optimize application performance on Copper.

The routines in LibSci are automatically included when using the ftn, cc, or CC commands. You do not need to use the "-l sci" flag in your compile command line.

Cray LibSci includes the following:

  • Basic Linear Algebra Subroutines (BLAS) - Levels 1, 2, and 3
  • Linear Algebra Package (LAPACK)
  • Scalable LAPACK (ScaLAPACK) (distributed-memory parallel set of LAPACK routines)
  • Basic Linear Algebra Communication Subprograms (BLACS)
  • Iterative Refinement Toolkit (IRT)
  • SuperLU (for large, sparse nonsymmetrical systems of linear equations)
5.4.2. AMD Core Math Library (ACML)

In addition to LibSci, Copper also provides the AMD Core Math Library (ACML). ACML is a set of numerical routines tuned specifically for AMD64 platform processors. The routines, which are available via both FORTRAN and C interfaces, include the following:

The routines in the ACML can be accessed by loading the acml module. After acml is loaded, compiling with ftn will automatically reference the library. To load the acml module, use the module load command, as follows:

module load acml

5.4.3. Additional PET Libraries

There is also an extensive set of Math libraries available in the $PET_HOME/MATH directory on Copper. A list of PET and other commonly supplied tools, such as HDF5, CMake, and NetCDF can be found at

5.5. Debuggers

Copper provides the TotalView debugger and the DDT Debugger to assist users in debugging their code.

5.5.1. TotalView and DDT

TotalView is a debugger that supports threads, MPI, OpenMP, C/C++, and Fortran, mixed-language codes, advanced features like on-demand memory leak detection, other heap allocation debugging features, and the Standard Template Library Viewer (STLView). Unique features like dive, a wide variety of breakpoints, the Message Queue Graph/Visualizer, powerful data analysis, and control at the thread level are also available.

DDT is a debugger that supports threads, MPI, OpenMP, C/C++, and Fortran, Coarray Fortran, UPC, and CUDA. Memory debugging and data visualization are supported for large-scale parallel applications. The Parallel Stack Viewer is a unique way to see the program state of all processes and threads at a glance.


Follow the steps below to use TotalView on Copper via a UNIX X-Windows interface.

  1. Ensure that an X server is running on your local system. Linux users will likely have this by default, but MS Windows users will need to install a third party X Windows solution. There are various options available.
  2. For Linux users, connect to Copper using "ssh -Y". Windows users will need to use PuTTY with X11 forwarding enabled (Connection->SSH->X11->Enable X11 forwarding).
  3. Compile your program on Copper with the "-g" option.
  4. Submit an interactive job:

    qsub -l select=1:ncpus=32:mpiprocs=32 -A Project_ID -l walltime=00:30:00 -q debug -X -I

    Once your job has been scheduled, you will be logged into an interactive batch session on a service node that is shared with other users.

  5. Load the TotalView module:

    . /usr/share/modules/init/ksh
    module load totalview
  6. Start program execution:

    totalview aprun -a -n 4 ./my_mpi_prog.exe arg1 arg2 ...

  7. After a short delay, the TotalView windows will pop up. Click "GO" and then "Yes" to start program execution.

An example of using TotalView can be found in $SAMPLES_HOME/Programming/Totalview_Example on Copper. For more information on using TotalView, see the TotalView Documentation page.


To use DDT on Copper, follow steps 1 through 4 as for TotalView above, but load and use the DDT debugger instead.

  1. Load the DDT module:

    . /usr/share/modules/init/ksh
    module load ddt

  2. Start program execution:

    ddt -n 4 ./my_mpi_prog.exe arg1 arg2 ...

  3. The DDT window will pop up. Verify the application name and number of MPI processes. Click "Run".

An example of using DDT can be found in $SAMPLES_HOME/Programming/DDT_Example on Copper. For more information on using DDT, see the DDT User Guide.

5.6. Code Profiling and Optimization

Profiling is the process of analyzing the execution flow and characteristics of your program to identify sections of code that are likely candidates for optimization, which increases the performance of a program by modifying certain aspects for increased efficiency.

We provide CrayPat to assist you in the profiling process. In addition, a basic overview of optimization methods with information about how they may improve the performance of your code can be found in Performance Optimization Methods (below).

5.6.1. CrayPat

CrayPat is an optional performance analysis tool used to evaluate program behavior on Cray supercomputer systems. CrayPat consists of the following major components: pat_build, pat_report, and pat_help. The data produced by CrayPat also can be used with Cray Apprentice2, an analysis tool that is used to visualize and explore the performance data captured during program execution.

Man pages are available for pat_build, pat_report, pat_help, and Apprentice2. Additional information can be found in the document "Using Cray Performance Analysis Tools."

The following steps should get you started using CrayPat:

  1. Load the "perftools" module

    module load perftools

  2. Compile the code, creating object files.

    ftn mycode.f90 -c

  3. Link the object files into your executable.

    ftn *.o -o mycode.x

  4. Use the pat_build command to generate an instrumented executable.

    pat_build -g mpi -u mycode.x mycode.x+pat

    This generates an instrumented executable called mycode+pat. Here the "-g" option enables the "mpi" tracegroup. See "man pat_build" for available tracegroups.

  5. Run the instrumented executable with aprun via PBS.

    aprun -n 4 ./mycode+pat

    This generates an instrumented output file (e.g., mycode+pat+2007-12tdt.xf).

  6. Use pat_report to display the statistics from the output file

    pat_report mycode+pat+2007-12tdt.xf > mycode.pat_report

Additional profiling options are available. See "man pat_build" for additional instrumentation options.

5.6.2. Additional Profiling Tools

There is also a set of profiling tools available in the $PET_HOME/pkgs directory on Copper. Information about these tools may be found on the Baseline Configuration Web site at BC policy FY13-01.

5.6.3. Program Development Reminders

If an application is not programmed for distributed memory, then only the cores on a single node can be used. This is limited to 32 cores on Copper.

Keep the system architecture in mind during code development. For instance, if your program requires more memory than is available on a single node, then you will need to parallelize your code so that it can function across multiple nodes.

5.6.4. Compiler Optimization Options

The "-Olevel" option enables code optimization when compiling. The level that you choose (0-4) will determine how aggressive the optimization will be. Increasing levels of optimization may increase performance significantly, but you should note that a loss of precision may also occur. There are also additional options that may enable further optimizations. The following table contains the most commonly used options.

Compiler Optimization Options
Option Description Compiler Suite
-O0 No Optimization. (default in GNU) All
-O1 Scheduling within extended basic blocks is performed. Some register allocation is performed. No global optimization. All
-O2 Level 1 plus traditional scalar optimizations such as induction recognition and loop invariant motion are performed by the global optimizer. Generally safe and beneficial. (default in PGI, Cray, & Intel) All
-O3 Levels 1 and 2 plus more aggressive code hoisting and scalar replacement optimizations that may or may not be profitable. Generally beneficial. All
-O4 Levels 1, 2, and 3 plus hoisting of guarded invariant floating point expressions is enabled. PGI
Chooses generally optimal flags for the target platform. Includes: -O2 -Munroll=c:1 -Mnoframe -Mlre -Mautoinline -Mvect=sse -Mscalarsse -Mcache_align -Mflushz. PGI
-Mipa=fast,inline Performs Interprocedural Analysis (IPA) with generally optimal IPA flags for the target platform, and inlining. IPA can be very time-consuming. Flag must be used in both compilation and linking steps. PGI
Minline=levels:n Number of levels of inlining (default: n = 1) PGI
-fipa-* The GNU compilers automatically enable IPA at various -O levels. To set these manually, see the options beginning with -fipa in the gcc man page. GNU
-O ipan Specifies various levels of inlining (n=0-5) Cray
-O vectorn Specifies various levels of vectorization (n = 0-3) Cray
-finline-functions Enables function inlining within a single file Intel
-ipon Enables interprocedural optimization between files and produces up to n object files Intel
-inline-level=n Number of levels of inlining (default: n=2) Intel
-ra Creates a listing file with optimization info Cray
-Mlist Creates a listing file with optimization info PGI
-Minfo Info about optimizations performed PGI
-Mneginfo Info on why certain optimizations are not performed PGI
-opt-reportn Generate optimization report with n levels of detail Intel
5.6.5. Performance Optimization Methods

Optimization generally increases compilation time and executable size, and may make debugging difficult. However, it usually produces code that runs significantly faster. The optimizations that you can use will vary depending on your code and the system on which you are running.

Note: Before considering optimization, you should always ensure that your code runs correctly and produces valid output.

In general, there are four main categories of optimization:

  • Global Optimization
  • Loop Optimization
  • Interprocedural Analysis and Optimization(IPA)
  • Function Inlining
Global Optimization

A technique that looks at the program as a whole and may perform any of the following actions:

  • Performed on code over all its basic blocks
  • Performs control-flow and data-flow analysis for an entire program
  • Detects all loops, including those formed by IF and GOTOs statements and performs general optimization.
  • Constant propagation
  • Copy propagation
  • Dead store elimination
  • Global register allocation
  • Invariant code motion
  • Induction variable elimination
Loop Optimization

A technique that focuses on loops (for, while, etc.) in your code and looks for ways to reduce loop iterations or parallelize the loop operations. The following types of actions may be performed:

  • Vectorization - rewrites loops to improve memory access performance. Some compilers may also support automatic loop vectorization by converting loops to utilize low-level hardware instructions and registers if they meet certain criteria.
  • Loop unrolling - (also known as "unwinding") replicates the body of loops to reduce loop branching overhead and provide better opportunities for local optimization.
  • Parallelization - divides loop operations over multiple processors where possible.
Interprocedural Analysis and Optimization (IPA)

A technique that allows the use of information across function call boundaries to perform optimizations that would otherwise be unavailable.

Function Inlining

A technique that seeks to reduce function call and return overhead.

  • Used with functions that are called numerous times from relatively few locations.
  • Allows a function call to be replaced by a copy of the body of that function.
  • May create opportunities for other types of optimization
  • May not be beneficial. Improper use may increase code size and actually result in less efficient code.

6. Batch Scheduling

6.1. Scheduler

The Portable Batch System (PBS) is currently running on Copper. It schedules jobs and manages resources and job queues, and can be accessed through the interactive batch environment or by submitting a batch request. PBS is able to manage both single-processor and multiprocessor jobs. The PBS module is automatically loaded for you when you log in.

6.2. Queue Information

The following table describes the PBS queues available on Copper:

Queue Descriptions and Limits
Priority Queue
Max Wall
Clock Time
Max Cores
Per Job
Highest urgent Urgent 168 Hours 4,096 Designated urgent jobs
Down Arrow for decreasing priority test N/A 48 Hours N/A Staff-only testing
debug* Debug 1 Hour 512 User testing
frontier Frontier 168 Hours 4,096 Frontier projects only
high High 168 Hours 4,096 Designated high-priority jobs
by Service/Agency
standard Standard 168 Hours 4,096 Normal priority jobs
transfer N/A 48 Hours 1 Data transfer jobs. Access to the long-term storage.
Lowest background** Background 4 Hours 4,096 Unrestricted access - no allocation charge.

* The running job limit on the debug queue per user is 4.
** The running job limit on the background queue per user is 6.

6.3. Interactive Logins

When you log in to Copper, you will be running in an interactive shell on a login node. The login nodes provide login access for Copper and support such activities as compiling, editing, and general interactive use by all users. Please note the Login Node Abuse policy. The preferred method to run resource-intensive executions is to use Cluster Compatibility Mode from within an interactive batch session.

6.4. Interactive Batch Sessions

To get an interactive batch session, you must first submit an interactive batch job through PBS. This is done by executing a qsub command with the "-I" option from within the interactive login environment. For example:

qsub -l select=N1:ncpus=32:mpiprocs=N2 -A Project_ID -q queue_name -l walltime=HHH:MM:SS -I

You must specify the number of nodes requested (N1), the number of processes per node (N2), the desired maximum walltime, your project ID, and a job queue. Valid values for N2 are 1, 2, 4, 8, 16, and 32.

Your interactive batch sessions will be scheduled just as normal batch jobs are scheduled depending on the other queued batch jobs, so it may take quite awhile. Once your interactive batch shell starts, it will be running on a service node that is shared by other users. At this point, you can launch parallel applications onto your assigned set of compute nodes by using the aprun command. You can also run interactive commands or scripts on this service node, but you should limit your memory and cpu usage. Use the Cluster Compatibility Mode for executing memory- and process-intensive commands such as tar and gzip/gunzip and certain serial applications directly on a dedicated compute node.

6.5. Cluster Compatibility Mode (CCM)

You can also request direct access to a compute node by including the "ccm" option in your PBS interactive batch job submission. For example:

qsub -l ccm=1 -l select=#:ncpus=32:mpiprocs=32 -A Project_ID -q queue_name -l walltime=HHH:MM:SS -I

Once scheduled by the PBS scheduler, you will again have an interactive shell session on a shared service node. Then, issue the ccmlogin command, and you will be logged onto the first compute node in the set of nodes to which you have been assigned. Your environment will react much the same as a normal shared service node. However, you will now have dedicated access to the entire compute node which will allow you to run serial applications as well as memory- and process-intensive commands such as tar and gzip/gunzip without affecting other users.

6.6. Batch Request Submission

PBS batch jobs are submitted via the qsub command. The format of this command is:

qsub [ options ] batch_script_file

qsub options may be specified on the command line or embedded in the batch script file by lines beginning with "#PBS".

For a more thorough discussion of PBS Batch Submission on Copper, see the Copper PBS Guide.

6.7. Batch Resource Directives

Batch resource directives allow you to specify to PBS how your batch jobs should be run and what resources your job requires. Although PBS has many directives, you only need to know a few to run most jobs.

The basic syntax of PBS directives is as follows:

#PBS option[[=]value]

where some options may require values to be included. For example, to set the number of cores for your job, you might specify the following:

#PBS -l select=#:ncpus=32:mpiprocs=#

The following directives are required for all jobs:

Required Directives
Directive Value Description
-A Project_ID Name of the project
-q queue_name Name of the queue
-l select=N1:npcus=32:mpirocs=N2 N1 = Number of nodes
N2 = MPI processes per node
(N2 must be: 1, 2, 4, 8, 16, or 32)
-l walltime=HHH:MM:SS Maximum wall time

A more complete listing of batch Resource Directives is available in the Copper PBS Guide.

6.8. Launch Commands

On Copper the PBS batch scripts and the PBS interactive login session run on a service node, not a compute node. The only way to send your executable to the compute nodes is to use the aprun command. The following example command line could be used within your batch script or in a PBS interactive session, sending the executable ./a.out to 64 compute cores.

aprun -n 64 ./a.out

Other Common Options for Aprun
Option Description
-N # The number of MPI processes to place per node. Useful for getting more memory per MPI process.
-d # The number of threads per node, OpenMP

A serial executable can be sent to one compute node using aprun or ccmrun:

aprun -n 1 serial_executable ## OR
ccmrun serial_executable

It is also possible to run a script on one compute node using ccmrun when the Cluster Compatable Mode has been envoked (-l ccm=1).

ccmrun script_to_run

Use aprun to launch MPI, SHMEM, OpenMP, Hybrid MPI/OpenMP, and PGAS executables. For examples of this, see MPI, SHMEM, OpenMP, Hybrid MPI/OpenMP, and PGAS (above) or look in the $SAMPLES_HOME directory on Copper. For more information about aprun, see the aprun man page.

6.9. Sample Script

While it is possible to include all PBS directives at the qsub command line, the preferred method is to embed the PBS directives within the batch request script using "#PBS". The following script is a basic example and contains all of the required directives, some frequently used optional directives, and common script components. More thorough examples are available in the Copper PBS Guide and in the Sample Code Repository ($SAMPLES_HOME) on Copper.

## The first line (above) specifies the shell to use for parsing the
## remaining lines of the batch script.

## Required PBS Directives --------------------------------------
#PBS -A Project_ID
#PBS -q standard
#Use 64 cores requiring 2 nodes
#PBS -l select=2:ncpus=32:mpiprocs=32
#PBS -l walltime=12:00:00

## Optional PBS Directives --------------------------------------
#PBS -N Test_Run_1
#PBS -j oe
#PBS -S /bin/bash
#PBS -l ccm=1
## ccm=1 option for Cluster Compatability Mode, including using
## executables compiled with Dynamic Shared Libraries 

## Execution Block ----------------------------------------------
# Environment Setup
# cd to your personal directory in the scratch file system

# create a job-specific subdirectory based on JOBID and cd to it
JOBID=`echo ${PBS_JOBID} | cut -d '.' -f 1`
if [ ! -d ${JOBID} ]; then
  mkdir -p ${JOBID}
cd ${JOBID}

# Launching
# copy executable from $HOME and submit it
cp ${HOME}/my_prog.exe .
# run all 64 cores
aprun -B ./my_prog.exe > my_prog.out

# Clean up
# archive your results
# Using the "here document" syntax, create a job script
# for archiving your data.
rm -f archive_job
cat >archive_job <<END
##PBS -l walltime=12:00:00
##PBS -q transfer
##PBS -A Project_ID
##PBS -l select=1:ncpus=1
##PBS -j oe
##PBS -S /bin/bash
archive mkdir -C ${ARCHIVE_HOME} ${JOBID}
archive put -C ${ARCHIVE_HOME}/${JOBID} *.out
archive ls ${ARCHIVE_HOME}/${JOBID}

# Remove scratch directory from the file system.
rm -rf ${JOBID}
sed -i -e"s/##PBS/#PBS/g" archive_job

# Submit the archive job script.
qsub archive_job

6.10. PBS Commands

The following commands provide the basic functionality for using the PBS batch system:

qsub: Used to submit jobs for batch processing.
qsub [options] my_job_script

qstat: Used to check the status of submitted jobs.
qstat PBS_JOBID ## check one job
qstat -u my_user_name ## check all of user's jobs

qdel: Used to kill queued or running jobs.

A more complete list of PBS commands is available in the Copper PBS Guide.

6.11. Determining Time Remaining in a Batch Job

In batch jobs, knowing the time remaining before the workload management system will kill the job enables the user to write restart files or even prepare input for the next job submission. However, adding such capability to an existing source code requires knowledge to query the workload management system as well as parsing the resulting output to determine the amount of remaining time.

The DoD HPCMP allocated systems now have the library, WLM_TIME, as an easy way to provide the remaining time in the batch job to C, C++, and Fortran programs. The library can be added to your job using the following:

For C:

#include <wlm_time.h>
void wlm_time_left(long int *seconds_left)

For Fortran:

INTEGER seconds_left

For C++:

extern "C" {
#include <wlm_time.h>

For simplicity, wall-clock-time remaining is returned as an integer value of seconds.

To simplify usage, a module file defines the process environment, and a pkg-config metadata file defines the necessary compiler linker options:

For C:

module load wlm_time
$(CC) ctest.c `pkg-config --cflags --libs wlm_time`

For Fortran:

module load wlm_time
$(F90) test.f90 `pkg-config --cflags-only-I --libs wlm_time`

For C++:

module load wlm_time
$(CXX) Ctest.C `pkg-config --cflags --libs wlm_time`

WLM_TIME works currently with PBS. The developers expect that WLM_TIME will continue to provide a uniform interface encapsulating the underlying aspects of the workload management system.

6.12. Advance Reservations

An Advance Reservation Service (ARS) is available on Copper for reserving up to 3744 cores for use, starting at a specific date/time, and lasting for a specific number of hours. The ARS is accessible via most modern web browsers at Authenticated access is required. An ARS User's Guide is available online once you have logged in.

7. Software Resources

7.1. Application Software

All Commercial Off The Shelf (COTS) software packages can be found in the $CSI_HOME directory (/usr/local/applic). A complete listing of the software versions installed on Copper can be found on our software page. The general rule for all COTS software packages is that the two latest versions will be maintained on our systems. For convenience, modules are also available for most COTS software packages.

7.2. Useful Utilities

The following utilities are available on Copper:

Useful Utilities
archive Perform basic file-handling operations on the archive server
blocker Convert a file to fixed-record-length format
bull Display the system bulletin board
cal2jul Convert a date into the corresponding Julian day
datecalc Print an offset from today's date in various formats
extabs Expand tab characters to spaces in a text file
getarchome Display the value of $ARCHIVE_HOME for a given userid
getarchost Display the value of $ARCHIVE_HOST for a given userid
justify Justify a character string with padding
lss Show unprintable characters in file names
mpibzip2 Parallel implementation of the bzip2 compression utility
mpscp High-performance remote file copy
node_use Display the amount of free and used memory for login nodes
qlim Report current batch queue usages and limits.
qpeek Display spooled stdout and stderr for an executing batch job.
qview Display information about batch jobs and queues
show_queues Report current batch queue status, usage, and limits
show_resv Show status of advance reservations
show_storage Display archive server allocation and usage by subproject
show_usage Display CPU allocation and usage by subproject
stripdos Strip DOS end-of-record control characters from a text file
stripes Report the OST striping pattern of files in a Lustre file system
tails Display the last five lines of one or more files
trim Trim trailing blanks from text file lines
vman Browse an online man page using the view command

7.3. Sample Code Repository

The Sample Code Repository is a directory that contains examples for COTS batch scripts, building and using serial and parallel programs, data management, and accessing and using serial and parallel math libraries. The $SAMPLES_HOME environment variable contains the path to this area, and is automatically defined in your login environment. Users should look in the $SAMPLES_HOME directory for simple examples. Examples and useful tips are periodically added to this directory. If you have a suggestion, correction, or example of your own that would be useful to other users, submit it to the HPC Help Desk for consideration.

Sample Code Repository on Copper
Use of the application name resource.
application_namesREADME and list of valid strings for application names intended for use in every PBS script preamble. The HPCMP encourages applications not specifically named in the list to be denoted as "other".
Application-specific examples; interactive job submit scripts; use of the application name resource; software license use.
abaqusBasic batch script, README, and input deck for an ABAQUS job.
cobaltSample input files and job script for the COBALT application.
espressoSample input files, READMEs, and job script for the Quantum ESPRESSO application
fluentSample input files, README, and job script for the FLUENT application.
gaspexSample input files, READMEs, and job script for the GASP application.
gaussianSample README and job scripts for the GAUSSIAN application.
lammpsSample input file and job script for the LAMMPS application.
ls_dynaSample input file, README, and job script for the LS_DYNA application.
matlabSample input file and job script demonstrating appropriate interactive and queued job use of the MATLAB application.
nwchemSample input files, README, and job script for the NWCHEM application.
star-ccm+Sample input files, README, and job scripts for the STAR-CCM+ application.
vaspSample input files, README, and job script for the VASP application.
Archiving and retrieving files; Lustre striping; file searching; $WORKDIR use.
MPSCP_to_DMSInstructions and sample scripts on using the mpscp utility for transferring files to/from DMS.
OST_StripesInstructions and examples for striping large files on the Lustre file systems.
Postprocess_ExampleExample showing how to submit a post-processing script at the end of a parallel computation job to do such things as tar data and store it off of temporary storage to archive storage.
Transfer_Queue_ExampleTwo PBS batch script examples for data transfer to archive storage.
Transfer_Queue_with_Archive_CommandsExample and instructions on recommended best practice to stage data from mass storage using the transfer queue prior to job execution, then processing using that data, then passing output data back to mass storage using the transfer queue again.
MPI, OpenMP, and hybrid examples; single-core jobs; large memory jobs; running multiple applications within a single batch job.
CCMRUN_ExampleUsing CCM to run multiple executions on the compute nodes at the same time, and X11 forwarding with ccmlogin to a compute node.
Cluster_ModeUsing Cluster Compatibility Mode.
Hello_World_ExampleSample codes and job scripts for MPI, OpenMP and Hybrid MPI/OpenMP applications of hello world PBS jobs.
Multiple_Jobs_per_NodeRunning serial jobs in parallel on one node; running serial mixed with parallel; running multiple distinct parallel tasks; running a single parallel task with multiple different binaries.
Basic code compilation; debugging; use of library files; static vs. dynamic linking; Makefiles; Endian conversion.
BLACS_ExampleSample BLACS Fortran program, compile script and PBS submission scripts. The BLACS are in the ScaLAPACK library in $PET_HOME.
BLACS_w_ACML_ExampleSample BLACS Fortran program, compile script linking against AMD's Core Math Library (ACML), and PBS submission scripts. ACML is AMD's serial linear algebra library. The BLACS are in Cray's libsci.
CCM-DSL_ExampleCompile and run a dynamically linked executable with DSL and CCM support on compute nodes.
COMPILE_INFOInstructions for using the (default) PGI compilers to compile Fortran and C code using MPI. Various optimization flags are discussed. Includes the flags to compile OpenMP code for all programming environments. Also includes instructions for using the configure utility to a Makefile for any system.
Core_FilesInstructions and source code for viewing core files with different debuggers - TotalView, gdb.
CrayPat_ExampleSample exercise demonstrating how to use Cray's profiling tool Craypat to examine the performance of a parallel code. Includes discussion of the use of hardware performance counters.
DDT_ExampleUsing DDT to debug a small example code in an interactive batch job.
Endian_ConversionText file presenting the PGI, Cray, Gnu, and Intel compiler options for enabling binary data formatted on a different architecture to be readable by code compiled on Copper.
HDF5_ExampleShort sample codes in C, Fortran, and C++ for using HDF5 and Parallel HDF5 on Copper to efficiently process HDF5 codes.
Libsci_ExampleSample ScaLAPACK Fortran program, compile script and PBS submission scripts. Cray's libsci, the default and best optimized, are used.
Libsci_w_ACML_ExampleSample ScaLAPACK Fortran program, compile script linking against AMD's Core Math Library (ACML) and Cray's libsci. PBS submission scripts are included.
Link_LibrariesThe libraries for linking are different from similar Cray machines, in large part due to changes in X11 libraries.
Memory_UsageA routine callable from Fortran or C for determining how much memory a process is using.
Open_Files_LimitBrief presentation of the maximum number of simultaneously open files for a single process and for a user job. Makefile, code, and PBS scripts are included to enable verification.
ScaLAPACK_ExampleSample ScaLAPACK Fortran program, compile script and PBS submission scripts. Netlib's ScaLAPACK library in $PET_HOME and AMD's ACML are used.
SO_CompileSample Shared Object (ie, dynamically linked library, or DLL) compilation info.
Timers_FortranSerial Timers using Fortran Intrinsics f77 and f90/95.
TotalView_ExampleUsing TotalView to debug a small example code in an interactive batch job.
Use of modules; customizing the login environment; use of common environment variables to facilitate portability of work between systems
Module_Swap_ExampleBatch script demonstrating how to swap one module version for another.
Basic batch scripting; use of the transfer queue; job arrays; job dependencies; Secure Remote Desktop; job monitoring.
BatchScript_ExampleSimple PBS batch script showing all required options.
Core_Info_ExampleSample code for generating the MPI process/core or OpenMP thread/core associativity in compute jobs.
Hybrid_ExampleSample script for running hybrid MPI/OpenMP jobs.
Job_Array_ExampleSample job script for using job arrays.
Job_Dependencies_ExampleExample code, scripts, and instructions demonstrating how to set up a job dependency for jobs depending on how one or more other jobs execute, or to perform some action that one or more other jobs require before execution.
MPI_ExampleSample script for running MPI jobs.
OpenMP_ExampleSample script for running OpenMP jobs.
Shmem_ExampleSimple example of using SHMEM.
Transfer_QueuePBS batch script example for data transfer using the transfer queue.

8. Links to Vendor Documentation

Cray Home:
Cray Application Developer's Environment User's Guide

Novell Home:
Novell SUSE Linux Enterprise Server:

GNU Home:
GNU Compiler:
Portland Group Resources Page:
Portland Group User's Guide: