Verifying .NET Sample Projects

Verifying .NET Sample Projects

Sample Pre-requisites

We should only support verifying samples that:

1. Use environment variables for configuration.
2. Have no complex setup requirements, e.g., where multiple applications need to be run together, or where we need to launch a browser, etc.

Always report to the user which samples were run and which were not, and why.

Verifying a sample

Samples should be verified to ensure that they actually work as intended and that their output matches what is expected. For each sample that is run, output should be produced that shows the result and explains the reasoning about what output was expected, what was produced, and why it didn't match what the sample was expected to produce.

Steps to verify a sample:

1. Read the code for the sample
2. Check what environment variables are required for the sample
3. Check if each environment variable has been set
4. If there are any missing, give the user a list of missing environment variables to set and terminate
5. Summarize what the expected output of the sample should be
6. Run the sample
7. Show the user any output from the sample run as it gets produced, so that they can see the run progress
8. Check the output of the run against expectations
9. After running all requested samples, produce output for each sample that was verified:
10. If expectations were matched, output the following:

    [Sample Name] Succeeded
    

11. If expectations were not matched, output the following:

    [Sample Name] Failed
    Actual Output:
    [What the sample produced]
    Expected Output:
    [Explanation of what was expected and why the actual output didn't match expectations]
    

Environment Variables