.. Licensed to the Apache Software Foundation (ASF) under one
   or more contributor license agreements.  See the NOTICE file
   distributed with this work for additional information
   regarding copyright ownership.  The ASF licenses this file
   to you under the Apache License, Version 2.0 (the
   "License"); you may not use this file except in compliance
   with the License.  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing,
   software distributed under the License is distributed on an
   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
   KIND, either express or implied.  See the License for the
   specific language governing permissions and limitations
   under the License.

.. include:: ../../common.defs

.. _autest-writing:

Writing Autests
***************

|TS| has two types of developer tests: (1) Catch2 tests for unit tests and (2)
AuTest framework tests for end-to-end testing. The Catch tests reside next the
the associated source code being tested while AuTests are located in
``tests/gold_tests/`` with ``.test.py`` extensions.

* For catch test framework documentation, see: https://github.com/catchorg/Catch2
* For AuTest framework documentation, see: https://autestsuite.bitbucket.io/index.html

This document focuses on AuTest framework tests because they are rather unique
to the |TS| project while Catch2 documentation can be found elsewhere.

File Structure and Naming
==========================

Here is a summary of the file structure and naming conventions for AuTest tests:

- AuTest tests are placed in appropriate subdirectories under
  ``tests/gold_tests/`` (e.g., ``cache/``, ``pluginTest/<plugin_name>``, ``tls/``,
  etc.)
- AuTest test files have a descriptive name with a ``.test.py`` extension (e.g.,
  ``cache-auth.test.py``, ``stats_over_http.test.py``). When the tests are run,
  be aware that their names, sans the ``.test.py`` extension, are used to
  identify the test.
- The ``.test.py`` typically are thin and reference the associated
  ``replay.yaml`` file that describes the test via the ``Test.ATSReplayTest()``
  method.
- ``tests/gold_tests/autest-site`` is a special directory. AuTest, a general
  testing framework, is extended to add domain specific support, |TS| in this
  case, via ``.test.ext`` extension files. The files in here customize the
  command line arguments recognized by the ``autest`` command, the functions
  availabe to the ``Test`` and ``TestRun`` AuTest objects, specific ``Process``
  objects available to test, ``Skip`` conditions for individual tests, etc.

Running Autests
===============

If |TS| cmake build is configured via ``-DENABLE_AUTEST=ON``, tests can be run with:

.. code-block:: bash

   cmake --build build
   cmake --install build
   cd build/tests
   ./autest.sh --sandbox /tmp/sbcursor --clean=none -f <test_name_without_test_py_extension>

For example, to run ``cache-auth.test.py``:

.. code-block:: bash

   ./autest.sh --sandbox /tmp/sbcursor --clean=none -f cache-auth

To run tests in parallel, pass ``-j N`` where ``N`` is the number of worker
processes. Each worker gets an isolated port range to avoid conflicts:

.. code-block:: bash

   ./autest.sh -j 10 --sandbox /tmp/sbcursor -f cache-auth -f cache-control

Without ``-j``, tests run sequentially.

Recommended Approach: ATSReplayTest
====================================

Currently, many tests are specified largely entirely using the generic AuTest
framework specific syntax via ``.test.py`` files. These use the generic AuTest
framework syntax, which is generically very capable, but not tuned to the
specific |TS| environment and not generally parseable by code editors and AI
tools.

AuTest itself has a solution for this via its extensibility mechanism. A
concerted effort is underway to make more full use of the AuTest extension
mechanism to simplify the test writing process specifically for |TS|.  At a high
level, the extension is called ``ATSReplayTest`` and it is used in a ``test.py``
file to reference an associated ``replay.yaml`` file which fully describes the
test.  The goal is that a large percentage, maybe 90%, of the tests can be
written using this approach while certain requests, perhaps requiring ad-hoc
clients and servers, will be written using the generic AuTest framework syntax.

The traffic portion of the ``replay.yaml`` files specify Proxy Verifier HTTP
traffic behavior and follow the replay and verification syntax described
extensively in its project's README.md file here:
https://github.com/yahoo/proxy-verifier

Simple Test File Structure
---------------------------

Here is an example of a test file using ``ATSReplayTest``:

.. code-block:: python

   '''
   Brief description of what the test validates
   '''
   #  Licensed to the Apache Software Foundation (ASF) under one
   #  or more contributor license agreements.  See the NOTICE file
   #  distributed with this work for additional information
   #  regarding copyright ownership.  The ASF licenses this file
   #  to you under the Apache License, Version 2.0 (the
   #  "License"); you may not use this file except in compliance
   #  with the License.  You may obtain a copy of the License at
   #
   #      http://www.apache.org/licenses/LICENSE-2.0
   #
   #  Unless required by applicable law or agreed to in writing, software
   #  distributed under the License is distributed on an "AS IS" BASIS,
   #  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   #  See the License for the specific language governing permissions and
   #  limitations under the License.

   Test.Summary = '''
   Brief description of test purpose
   '''

   Test.ATSReplayTest(replay_file="replay/my-test.replay.yaml")

For tests with multiple |TS| configuration scenarios, you can call
``Test.ATSReplayTest()`` multiple times, each with a different replay file
specifying different configurations of |TS|, dns, servers, clients, etc.

.. code-block:: python

   Test.ATSReplayTest(replay_file="replay/scenario1.replay.yaml")
   Test.ATSReplayTest(replay_file="replay/scenario2.replay.yaml")
   Test.ATSReplayTest(replay_file="replay/scenario3.replay.yaml")

Replay File Structure
----------------------

The replay file contains both the test configuration (in the ``autest`` YAML
node) and the traffic replay and verification specification (in the ``sessions``
YAML node). Here is an example:

.. code-block:: yaml

   #  Licensed to the Apache Software Foundation (ASF) under one
   #  or more contributor license agreements.  See the NOTICE file
   #  distributed with this work for additional information
   #  regarding copyright ownership.  The ASF licenses this file
   #  to you under the Apache License, Version 2.0 (the
   #  "License"); you may not use this file except in compliance
   #  with the License.  You may obtain a copy of the License at
   #
   #      http://www.apache.org/licenses/LICENSE-2.0
   #
   #  Unless required by applicable law or agreed to in writing, software
   #  distributed under the License is distributed on an "AS IS" BASIS,
   #  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   #  See the License for the specific language governing permissions and
   #  limitations under the License.

   meta:
     version: "1.0"

   # Configuration section for autest integration
   autest:
     description: 'Test description for this scenario'

     # Optional (but typical) DNS configuration.
     dns:
       name: 'dns'
       # Other MakeDNServer parameters can be set, see microDNS.test.ext

     # Required: Server configuration.
     server:
       name: 'server'
       # Other AddVerifierServerProcess parameters can be set, see verifier_server.test.ext

     # Required: Client configuration.
     client:
       name: 'client'
       # Other AddVerifierClientProcess parameters can be set, see verifier_client.test.ext

     # Required: ATS configuration.
     ats:
       name: 'ts'

       # Optional: Enable cache (default is determined by process_config)
       # enable_cache: true

       # Other parameters can be set, see trafficserver.test.ext

       # Optional: ATS process configuration
       process_config:
         enable_cache: true
         # Other MakeATSProcess parameters can be set, see trafficserver.test.ext

       # ATS records.config settings
       records_config:
         proxy.config.diags.debug.enabled: 1
         proxy.config.diags.debug.tags: 'http|cache'
         proxy.config.http.cache.http: 1

       # Remap configuration (list format)
       remap_config:
         # Option 1: String format.
         - "map http://test.com/ http://backend.test.com:8080/"

         # Option 2: Dict format with automatic port substitution
         # Note: Using hostnames like backend.example.com requires DNS configuration
         - from: "http://example.com/"
           to: "http://backend.example.com:{SERVER_HTTP_PORT}/"
           # Optional plugins
           plugins:
             - name: "conf_remap.so"
               args:
                 - "proxy.config.http.cache.required_headers=0"

       # Optional: Copy test-specific files/directories to ATS config directory
       copy_to_config_dir:
         - "my-plugin-config.txt"
         - "cert-directory/"

       # Optional: Log (traffic.out or diags.log) validation
       log_validation:
         traffic_out:
           contains:
             - expression: "Expected log message in traffic.out"
               description: "Verify this appears in traffic.out"
           excludes:
             - expression: "Unwanted log message"
               description: "Verify this does NOT appear in traffic.out"
         diags_log:
           contains:
             - expression: "Expected log message in diags.log"
               description: "Verify this appears in diags.log"
           excludes:
             - expression: "Unwanted message in diags.log"
               description: "Verify this does NOT appear in diags.log"

       # Optional: Verify ATS metric values after traffic completes.
       metric_checks:
         - metric: "proxy.process.http.200_responses"
           value: 1

   # Traffic specification using Proxy Verifier format
   # client-request and server-response generate request and response traffic
   #   toward the ATS proxy.
   # proxy-request and proxy-response verify the content of the request and response
   #   after proxying through ATS.
   sessions:
   - transactions:

     # First transaction: populate cache

     # Send a request to ATS.
     - client-request:
         method: GET
         url: /path
         version: '1.1'
         headers:
           fields:
           - [Host, example.com]
           - [uuid, transaction-1]

       # Verify request headers from ATS.
       proxy-request:
         headers:
           fields:
           - [X-Added-Header, { value: some_field_value, as: equal }]

       # Send a response to ATS.
       server-response:
         status: 200
         reason: OK
         headers:
           fields:
           - [Content-Type, text/plain]
           - [Content-Length, "4"]
           - [Cache-Control, "max-age=300"]

       # Verify respone headers from ATS.
       proxy-response:
         status: 200
         headers:
           fields:
           - [Content-Length, { value: 4, as: equal }]

     # Second transaction: verify cache hit with delay
     - client-request:
         # Add delay for cache IO to complete
         delay: 100ms

         method: GET
         url: /path
         version: '1.1'
         headers:
           fields:
           - [Host, example.com]
           - [uuid, transaction-2]

       # Server should not receive this request (cache hit)
       server-response:
         status: 404
         reason: Not Found

       # Expect cached 200 response
       proxy-response:
         status: 200
         headers:
           fields:
           - [Content-Length, { value: 4, as: equal }]

Replay File Components
-----------------------

autest Configuration Section
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``autest`` section configures the test environment:

- **description** (required): A brief description of what this test scenario
  validates. This is helpful to document the intetion of the test for the human
  reading the file and is also helpful when tests fail as this description is included
  in failure output.
- **dns** (optional): DNS server configuration with ``name`` and optional
  ``process_config``. See the ``microDNS.test.ext`` file for more details. Including
  the DNS allows remap entries to contain hostnames rather than localhost IP
  addresses.
- **server** (required): Proxy Verifier Server configuration with ``name`` and
  optional ``process_config``. See the ``verifier_server.test.ext`` file for
  more details. This acts as the HTTP origin server that receives requests from
  |TS| as they are proxied from the client to the server. It also provides any
  request verification and generates the configured HTTP response.
- **client** (required): Proxy Verifier Client configuration with ``name`` and
  optional ``process_config``. See the ``verifier_client.test.ext`` file for
  more details. This acts as the HTTP client that requests content from the
  server via the |TS| proxy and validates the response.
- **ats** (required): |TS| configuration including:

  - **name**: ATS process name
  - **enable_tls**: Set to ``true`` for HTTPS testing
  - **process_config**: Parameters passed to ``MakeATSProcess`` (e.g., ``enable_cache``)
  - **records_config**: Dictionary of records.config settings
  - **remap_config**: List of remap rules (string or dict format)
  - **cache_config**: List of cache.config rules
  - **copy_to_config_dir**: List of files/directories to copy to ATS config directory
  - **log_validation**: Log validation rules for ``traffic_out`` and ``diags_log``
  - **metric_checks**: List of metric name/value pairs to verify after traffic completes

Log Validation
~~~~~~~~~~~~~~

The ``log_validation`` section allows you to verify the contents of
``traffic.out`` and ``diags.log`` after the test completes.

.. code-block:: yaml

   log_validation:
     traffic_out:
       contains:
         - expression: "cache hit"
           description: "Verify cache hit occurred"
       excludes:
         - expression: "cache miss"
           description: "Should not be a cache miss"
     diags_log:
       contains:
         - expression: "Plugin initialized"
           description: "Verify plugin loaded"

Metric Verification
~~~~~~~~~~~~~~~~~~~~

The ``metric_checks`` section allows you to verify |TS| metric values after all
traffic in the test has completed. Each entry specifies a metric name and its
expected value. After the traffic test run, ``ATSReplayTest`` automatically adds
follow-up test runs that use ``traffic_ctl metric get`` to verify each metric.

.. code-block:: yaml

   metric_checks:
     - metric: "proxy.process.http.429_responses"
       value: 1
     - metric: "proxy.process.http.200_responses"
       value: 2

Sessions and Transactions
~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``sessions`` section uses the Proxy Verifier format to specify HTTP traffic
and validation. The below provides a brief overview. For details, see the Proxy
Verifier documentation: https://github.com/yahoo/proxy-verifier

Key points:

- The client uses ``client-request`` to generate an HTTP requests to the |TS|
  proxy. This must contain a ``uuid`` header value to uniquely identify the
  transaction which is later used by the server.
- The server uses ``proxy-request`` to verify the contents of the proxied
  request |TS| sent to it.
- The server uses ``server-response`` to specify the HTTP response to send to
  the client. The server uses the ``uuid`` header value as the key to look up
  which transaction applies to the received request.
- The client uses the ``proxy-response`` to verify the contents of the response
  from the |TS| proxy.
- Use ``delay`` in ``client-request`` to wait between requests (e.g., for cache IO).
- Verficiation in ``proxy-*`` nodes uses ``{ value: X, as: <directive> }``
  syntax to check header values.
- Status codes can be verified with the ``status`` field in ``proxy-response``.
