Testing E3SM Diagnostics
==============================================

Unit and integration tests
--------------------------

Run all automated tests by doing the following:

    .. code::

        pip install . # Install your changes
        ./tests/test.sh # Run all unit and integration tests


If these tests pass, you're done. If they fail unexpectedly however,
your code might have changed what the output looks like.

``ls tests/integration/image_check_failures``
will show you all the images that differ from expected.

If you see unexpected images listed, you'll have to investigate why your
code changed what the images look like.

If the only images listed are ones you expected to see changed,
then you need to update the expected images.


    .. code::

        cd /lcrc/group/e3sm/public_html/e3sm_diags_test_data/integration/expected
        cat README.md
        # This will show you the version, date, and hash of the current expected images.
        # Using that information do the following:
        mv integration_test_images previous_output/integration_test_images_<version>_<date>_<hash>
        # `cd` back into your E3SM Diags directory.
        # Your output will now become the new expectation.
        mv all_sets_results /lcrc/group/e3sm/public_html/e3sm_diags_test_data/integration/expected/integration_test_images
        cd /lcrc/group/e3sm/public_html/e3sm_diags_test_data/integration/expected

Run ``./tests/test.sh`` again. Now, the test should pass.

After merging your pull request, edit ``README.md``.
The version should be the version of E3SM Diags you ran ``./tests/test.sh`` with,
the date should be the date you ran ``./tests/test.sh`` on,
and the hash should be for the top commit shown by ``git log`` or on
https://github.com/E3SM-Project/e3sm_diags/commits/master.


Automated tests
---------------

We have a :ref:`GitHub Actions <ci-cd>` Continuous Integration / Continuous Delivery (CI/CD) workflow.

The unit and integration tests are run automatically as part of this.

Complete run test
-----------------

``tests/complete_run.py`` checks the images generated by all diagnostics to
see if any differ from expected.
This test is not run as part of the unit test suite, because it relies on a large
quantity of data found on LCRC (Anvil/Chrysalis).

    .. warning::
        You have to run this test manually. It is not run as part of the CI/CD workflow.

If you've been developing code on a different machine, you can get the code on a LCRC
machine by doing the following. First, push your code to GitHub.
Then log into a LCRC machine and run:

    .. code::

        git fetch <fork-name> <branch-name> # Fetch the branch you just pushed
        git checkout -b run-lcrc-test <repo-name>/<branch-name>

Now that you have your changes on LCRC, enter your development environment. Then:

    .. code::

        pip install . # Install your changes
        python -m unittest tests/complete_run.py

If this test passes, you're done. If it fails however, that means
your code has changed what the output looks like.

``ls image_check_failures`` will show your all the images that differ from expected.

If you see unexpected images listed, you'll have to investigate why your
code changed what the images look like.

If the only images listed are ones you expected to see changed,
then you need to update the expected images.


    .. code::

        cd /lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/expected
        cat README.md
        # This will show you the version, date, and hash of the current expected images.
        # Using that information do the following:
        mv all_sets previous_output/all_sets_<version>_<date>_<hash>
        mv image_list_all_sets.txt previous_output/image_list_all_sets_<version>_<date>_<hash>.txt
        # `cd` back into your E3SM Diags directory.
        # Your output will now become the new expectation.
        mv <version>_all_sets/ /lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/expected/all_sets
        cd /lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/expected/all_sets
        # This file will list all the expected images.
        find . -type f -name '*.png' > ../image_list_all_sets.txt
        cd ..

Run ``python -m unittest tests/complete_run.py`` again. Now, the test should pass.

After merging your pull request, edit ``README.md``.
The version should be the version of E3SM Diags you ran
``python -m unittest tests/complete_run.py`` with,
the date should be the date you ran ``python -m unittest tests/complete_run.py`` on,
and the hash should be for the top commit shown by ``git log`` or on
https://github.com/E3SM-Project/e3sm_diags/commits/master.