Defining tests¶
To execute a test, you will need:
A “test plan”, defined via one or more
*.yaml
files. Here’s an example:type: test/samples schema_version: 1 test: suites: - name: "Language samples test" setup: # can have yaml and/or code, just as in the cases below - code: log('In setup "hi"') teardown: # can have yaml and/or code, just as in the cases below - code: log('In teardown bye') cases: - name: "A test defined via yaml directives" spec: - log: - 'Reading from manifest at {language_analyze_sentiment_text:@manifest_source}' - call: sample: "language_analyze_sentiment_text" params: content: literal: "happy happy smile @hope" - assert_success: [] # try assert_failure to see how failure looks - assert_contains: - message: "Have score and magnitude" - literal: "score" - literal: "magnitude" - assert_contains_any: - message: "Have magnitude or strength" - literal: "strength" - literal: "magnitude" - assert_contains: - message: "Score is very positive" - literal: "score: 0.8" - assert_contains: - message: "Magnitude is very positive" - literal: "magnitude: 0.8" - assert_excludes: - message: "Random message" - literal: "The rain in Spain falls mainly in the plain" - assert_not_contains: # deprecated: use "assert_excludes" instead - message: "Random message" - literal: "The rain in Spain falls mainly in the plain"
See the Testplan page for information on the yaml directives available in the testplan, and how to use them directly via embedded Python code.
A “manifest”, defined via one or more
*.manifest.yaml
files. Here’s an example:--- type: manifest/samples schema_version: 3 samples: - environment: java invocation: "{jar} -D{class} {path} @args" path: "examples/mock-samples/java/language-v1/AnalyzeSentiment" class: AnalyzeSentiment jar: "./do_java" chdir: "examples/mock-samples/java/" sample: "language_analyze_sentiment_text" - environment: python bin: "python3" path: "examples/mock-samples/python/language-v1/analyze_sentiment_request_language_sentiment_text.py" sample: "language_analyze_sentiment_text" - environment: bash # notice: no "bin:" because artifacts are already executable path: "examples/mock-samples/sh/language-v1/analyze_sentiment.sh" sample: "language_analyze_sentiment_text"
See the Manifest file format page for an explanation of the manifest.