@@ -91,8 +91,6 @@ associated `Gen`erator, which `testgen` will invoke to instantiate the test
9191class. Its output will be a test file containing a point schedule and a
9292(mechanical) description of the property which needs to pass.
9393
94- <!-- TODO(sandy): how do we serialize properties? -->
95-
9694The ` runner ` tool accepts a test file (as output by ` testgen ` ) and
9795a * shrink index* (see @sec : shrinking ), and spins up simulated peers
9896corresponding to the embedded point schedule. The ` runner ` tool will then
@@ -112,24 +110,79 @@ can compare the final state of the NUT (as observed by the testing peer) and
112110ensure the desired property holds. Depending on the result of the test and the
113111state of the shrink index, we will perform different actions (see @sec : exit-codes .)
114112
113+ It is important to note that a single invocation of the composition `runner
114+ . testgen` correponds to a single unit test. Users are encouraged to run this
115+ composition in a loop, to gain the usual assurances given by property tests.
116+
117+ If a given test fails, and shrinking is desired, users can rerun ` runner ` with
118+ the outputted shrink index. The pair ` (testfile, shrinkindex) ` is the entirety
119+ of the shrink search state. By explicitly threading this state through flags
120+ and stdout, the system is stateless.
121+
122+
123+ ``` mermaid
124+ ---
125+ title: Basic Testing Workflow
126+ ---
127+ sequenceDiagram
128+
129+ participant PSG as Test Generator
130+ actor User
131+ participant Runner as Test Runner
132+ participant NUT as Node Under Test
133+
134+ Note over User,PSG: Choose test property
135+ User ->> PSG: testgen generate TESTCLASS
136+ PSG ->> User: file.test
137+
138+ loop for shrinkIndex
139+
140+ User ->> Runner: runner file.test shrinkIndex
141+ Runner ->> User: file.topology
142+ Note right of Runner: Runner spins-up<br/>simulated peers
143+ User ->> NUT: node --topology-file=file.topology
144+
145+ NUT -> Runner: Peer connection
146+ Runner -) NUT: Mock blocks
147+ NUT -) Runner: Diffused messages
148+ Runner -) NUT: ...
149+ Note over Runner,NUT: Simulation ends
150+
151+ Note right of Runner: Runner observes<br/>final state
152+ Note right of Runner: Runner evaluates<br/>test property
153+
154+ alt testBit exit_code KEEP_SHRINKING
155+ Runner ->> User: shrinkIndex
156+
157+ end
158+
159+ end
160+
161+ alt testBit exit_code TEST_FAILED
162+ Runner ->> User: minimal-couterexample
163+ else
164+ Note over Runner,NUT: Test passed
165+ end
166+ ```
167+
115168The ` shrinkView ` tool accepts a test file and a shrink index, and outputs the
116169test file corresponding to the given shrink index. This tool is primarily
117170useful for looking at non-minimal test inputs, eg, when the user doesn't want
118171to iterate the shrinking all the way down to a minimal example.
119172
120173
121- ### Supported Operations and Flags
174+ #### Supported Operations and Flags
122175
123176The ** test generator** CLI tool supports, at least, the following operations:
124177
125178- ` list-classes ` to list all available test classes.
126179- ` generate ` to produce a test file for a test class.
127- This operation has the following optional flags:
180+ This operation has the following optional flags:
128181 - ` --seed ` to specify a seed for the generator.
129182- ` meta ` to access test class metadata, eg the number of ` desired-passes `
130183 we expect to run a test for.
131184
132- On the other hand, the ** test runner** CLI tool supports the following optional flags:
185+ The ** test runner** CLI tool supports the following optional flags:
133186
134187- ` --shrink-index ` to specify the shrunk point schedule to run, according to the
135188 given index. The index values are output by the ` runner ` itself.
@@ -142,7 +195,7 @@ workflow. For example, users are free to run the entire test suite by looping
142195over ` testgen list-classes ` .
143196
144197
145- ### Exit Codes
198+ #### Exit Codes
146199
147200The ` Exit ` bit mask enum is used in the following section:
148201
0 commit comments