naming scheme for .input/.output test files

IKL Development
#1

: I'm hoping you have some thoughts here.


If I understand correctly, at some point, all of the .input files will be run against .ind scripts and validated against corresponding .output files, all as part of a test suite, presumably using command-runner.


At this point, I'm assuming the automated test suite will use file names to correlate .ind with .input/.output. So, e.g., get-system-status.ind will have get-system-status.input and get-system-status.output, and that's how the test suite will "know" which .input/.output files to run against a given .ind script.


So, then, will it possible to have multiple .input/.output pairs for one .ind script? (I hope so.) If so, how will we name them? e.g.,


get-system-status_xxxx.input/.output


Could the test suite pick that up and run it against the get-system-status.ind script? The current use case is fortigate, where we already have varying command output for the same command across multiple versions. We _could_ just jam all of this different output into a single .input file, but that could get really confusing -- especially for people coming later to the game....

#2

This is a great question. YES! In fact the more variations of outputs (.input) we can have, the better. Traditionally, one of the requirements to get a pull request approved, an IKE (Indeni Knowledge Expert) needed to have one .ind, one .output, and one .input. This is done for historical reference and because it's easy enough to manipulate the output variations manually. Afterall, command runner is a manually run test! Also, if you are a domain expert and know what those variations could be....easy peasy! But the key point you bring up here is automation. Automated tests will not be able to know what all possible outcomes are so it is essential to have several, if not many .input files to incorporate.


To answer your questions:

1. yes, it is okay to have more than one .input file in a pull request

2. yes, we can and will jam them all into one folder.

3. naming convention..this is another good question which has invited discussion, but no definitive corrrect answer. Ideas welcome! I would think that there is a clever way to comment inside the file to indicate possible scenario so that the name of the file is not as critical. That said all files should begin the same so that they are easily matchable to the .ind script by eye.


Hope this helps.


#3

We are working on a new Continuous Integration flow in which we will run ind against Input/Output. I'll creates a blog post once we have a working POC (should be on a couple of weeks)