Quick Start¶
This is a guide for a new user on a system thats already been properly setup. New users please referece to guide Installation.
Processflow install¶
You will need to first create an anaconda environment with the depedencies and install the processflow. Once conda has install all the python modules, create a run configuration file from the default and edit it to suit your case. You can find an Example Config here.
This install command will create a new anaconda environment with the latest stable release.
conda create --name processflow -c conda-forge -c e3sm -c cdat/label/v81 processflow
conda activate processflow
Use the following command to get the latest nightly build.
conda create --name processflow_nightly -c conda-forge -c e3sm/label/nightly -c e3sm -c cdat/label/v81 processflow
conda activate processflow_nightly
Once you have the processflow installed, the next step is creating your run configuration file. See Run Configuration on how to setup your run config.
Configuration¶
See here for real life configuration examples
Get a copy of the example config, and edit its keys to fit your case. You can get a copy of the example here:
wget https://raw.githubusercontent.com/E3SM-Project/processflow/master/run.cfg
For a complete explanation of each config key, see the Run Configuration guide.
For an example config with all the available options turned on see the Example Config
Execution¶
Running the processflow is simple. Once your config is setup, simply execute the following command:
processflow.py run.cfg
Once diagnostic output has been created, it will be moved to the host location (if img hosting is turned on). The web directories on NERSC and LLNL machines are password protected, credentials to view the output can be found here: https://acme-climate.atlassian.net/wiki/spaces/ATM/pages/41353486/How+to+run+AMWG+diagnostics+package?preview=%2F41353486%2F42730119%2Fcredentials.png
Machine Specific Configuration¶
Each of the E3SM supported machines has a slightly different setup for where they store input files and host output.
LCRC (Blues/Anvil)¶
All jobs should have the following keys added to their config when running on blues:
[[[custom_args]]]
--partition = acme-centos7
--account = condo
[global]
project_path = /lcrc/group/acme/<YOUR_USERNAME>/<YOUR_PROJECT>
[img_hosting]
img_host_server = web.lcrc.anl.gov
host_directory = /lcrc/group/acme/public_html/diagnostic_output/<YOUR_USERNAME>
url_prefix = /public/e3sm/diagnostic_output/<YOUR_USERNAME>
[post-processing]
[[climo]]
regrid_map_path = ~zender/data/maps/<YOUR_REGRID_MAP>
[diags]
[[e3sm_diags]]
reference_data_path = /lcrc/group/acme/diagnostics/obs_for_e3sm_diags/climatology/
[[amwg]]
diag_home = /lcrc/group/acme/amwg/amwg_diag
[[aprime]]
aprime_code_path = /lcrc/group/acme/diagnostics/a-prime/code
LLNL (acme1/aims4)¶
[global]
project_path = /p/user_pub/e3sm/<YOUR_USERNAME>/<YOUR_PROJECT>
[img_hosting]
img_host_server = acme-viewer.llnl.gov
host_directory = /var/www/acme/acme-diags/<YOUR_USERNAME>/
url_prefix = <YOUR_USERNAME>
[post-processing]
[[climo]]
regrid_map_path = /export/zender1/data/maps/<YOUR_REGRID_MAP>
[diags]
[[e3sm_diags]]
reference_data_path = /p/cscratch/acme/data/obs_for_acme_diags
[[amwg]]
diag_home = /p/cscratch/acme/amwg/amwg_diag
[[aprime]]
aprime_code_path = /p/cscratch/acme/data/a-prime
NERSC (cori)¶
NOTE: All jobs here need to be submitted to the “regular” partition, using the “e3sm” account.
[[[custom_args]]]
--partition = regular
--account = acme
-C = haswell
[global]
project_path = /global/project/projectdirs/acme/<YOUR_USERNAME>/<YOUR_PROJECT>
[img_hosting]
img_host_server = portal.nersc.gov
host_directory = /project/projectdirs/acme/www/<YOUR_USERNAME>
url_prefix = project/acme/<YOUR_USERNAME>
[post-processing]
[[climo]]
regrid_map_path = ~zender/data/maps/<YOUR_REGRID_MAP>
[diags]
[[e3sm_diags]]
reference_data_path = /global/project/projectdirs/acme/acme_diags/obs_for_e3sm_diags
[[amwg]]
diag_home = /global/project/projectdirs/acme/diagnostics/amwg/amwg_diag
[[aprime]]
aprime_code_path = <you have to clone the a-prime repo and set it up yourself>