Understanding the Performance Metrics

Autosubmit computes metrics to measure experiment performance. Besides the usual metrics, Simulated Years Per Day (SYPD), Core Hours per Simulated Year (CHSY), and Joules Per Simulated Year (JPSY), it also calculates two variations of SYPD, Actual Simulated Years Per Day (ASYPD) and Real Simulated Years Per Day (RSYPD).

Metrics Definition

Each experiment has defined a Parallelization variable (estimation is defined below), \(k\), which is the number of requested Processing Elements (PE), and the number of simulated years parameter, \(y\), calculated from CHUNK_SIZE, \(c_s\), and CHUNK_UNIT.

• If CHUNK_UNIT = year, \(y = c_s\)

• If CHUNK_UNIT = month, \(y = \frac{c_s}{12}\)

• If CHUNK_UNIT = day, \(y = \frac{c_s}{365}\)

• If CHUNK_UNIT = hour, \(y = \frac{c_s}{8760}\)

Let \(I\) be the set of SIM jobs whose state is COMPLETED. For each job \(i \in I\) the following attributes are taken into account:

• time in seconds in queue, \(q_i\)

• runtime in seconds, \(r_i\)

• energy consumption in joules, \(\eta_i\)

Let \(C\) be the set of SIM jobs that are COMPLETED. Let \(P\) be the set of POST jobs that are COMPLETED.

Experiment’s Actual Simulated Years Per Day: \(ASYPD = \frac{\sum_{i \in C} y_i \cdot 86400}{\sum_{i \in C} (q_i + r_i) + \frac{1}{|P|}\sum_{j \in P}(q_j+r_j)}\)

The computation of Real Simulated Year Per Day will only consider experiments that have a TRANSFER or a CLEAN job at the end of the workflow. AutosubmitAPI checks when the first SIM job started (not submitted), and when the last TRANSFER job finished. If the workflow does not have a TRANSFER job, then it checks the last CLEAN job finish time.

Let \(t_s\) be the time, in seconds, that the first SIM job started. Let \(t_f\) be the time in which the last TRANSFER or CLEAN job ended.

Experiment’s Real Simulated Years Per Day: \(RSYPD = \frac{\sum_{i \in C} y_i \cdot 86400}{t_f - t_s}\)

Generalization of SYPD and ASYPD

AutosubmitAPI can also compute SYPD and ASYPD for any job that has a chunk value. These jobs share the same attributes as SIM jobs. Only COMPLETED jobs are considered.

Generalized Simulated Years Per Day: \(SYPD_i = \frac{y_i \cdot 86400}{r_i}\)

As for ASYPD, it will only consider experiments that have at least one POST job.

Generalized Actual Simulated Years Per Day: \(ASYPD_i = \frac{y_i \cdot 86400}{(q_i + r_i) + \frac{1}{|P|}\sum_{j \in P}(q_j+r_j)}\)

Parallelization estimation

if NODES is None:
    if TASKS is not None and PROCESSORS != TASKS:
        NODES = ROUNDUP(PROCESSORS/TASKS)
        if PROCESSORS_PER_NODE is not None:
            PEs = NODES * PROCESSORS_PER_NODE
        else:
            ERROR(MISSING PROCESSORS_PER_NODE)
    elif PROCESSORS_PER_NODE is not None and PROCESSORS > PROCESSORS_PER_NODE:
        PEs = NHMPN(PROCESSORS)
    else:
        PEs = PROCESSORS  # PEs could not be totally accurate but we don't know how many PROCESSORS_PER_NODE
else:
    if NODES > 1 and PROCESSORS_PER_NODE is not None:
        PEs = NODES * PROCESSORS_PER_NODE
    elif NODES > 1:
        ERROR(MISSING PROCESSORS_PER_NODE)
    else:
        PEs = PROCESSORS