Defining tests

To execute a test, you will need:

  1. A “test plan”, defined via one or more *.yaml files. Here’s an example:

    language.test.yaml
    type: test/samples
schema_version: 1
test:
  suites:
  - name:  "Language samples test"
    setup:     # can have yaml and/or code, just as in the cases below
      - code:
          log('In setup "hi"')
    teardown:  # can have yaml and/or code, just as in the cases below
      - code:
          log('In teardown bye')          
    cases:
      
    - name: "A test defined via yaml directives"
      spec:
      - log:
          - 'Reading from manifest at {language_analyze_sentiment_text:@manifest_source}'
      - call:
          sample: "language_analyze_sentiment_text"
          params:
            content:
              literal: "happy happy smile @hope"
      - assert_success: [] # try assert_failure to see how failure looks
      - assert_contains:
          - message: "Have score and magnitude"
          - literal: "score"
          - literal: "magnitude"
      - assert_contains_any:
          - message: "Have magnitude or strength"
          - literal: "strength"
          - literal: "magnitude"
      - assert_contains:
          - message: "Score is very positive"
          - literal: "score: 0.8"
      - assert_contains:
          - message: "Magnitude is very positive"
          - literal: "magnitude: 0.8"
      - assert_excludes:
          - message: "Random message"
          - literal: "The rain in Spain falls mainly in the plain"
      - assert_not_contains:        # deprecated: use "assert_excludes" instead
          - message: "Random message"
          - literal: "The rain in Spain falls mainly in the plain"

    See the Testplan page for information on the yaml directives available in the testplan, and how to use them directly via embedded Python code.

  2. A “manifest”, defined via one or more *.manifest.yaml files. Here’s an example:

    language.manifest.yaml
    ---
type: manifest/samples
schema_version: 3
samples:
- environment: java
  invocation: "{jar} -D{class} {path} @args"
  path: "examples/mock-samples/java/language-v1/AnalyzeSentiment"
  class: AnalyzeSentiment
  jar: "./do_java"
  chdir: "examples/mock-samples/java/"
  sample: "language_analyze_sentiment_text"
- environment: python
  bin: "python3"
  path: "examples/mock-samples/python/language-v1/analyze_sentiment_request_language_sentiment_text.py"
  sample: "language_analyze_sentiment_text"
- environment: bash
  # notice: no "bin:" because artifacts are already executable
  path: "examples/mock-samples/sh/language-v1/analyze_sentiment.sh"
  sample: "language_analyze_sentiment_text"

    See the Manifest file format page for an explanation of the manifest.

References: