Skip to content

Commit 72c4c13

Browse files
authored
Insert diagram, add a note about statelessness (#31)
1 parent 1e65d12 commit 72c4c13

File tree

1 file changed

+59
-6
lines changed

1 file changed

+59
-6
lines changed

docs/design.md

Lines changed: 59 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -91,8 +91,6 @@ associated `Gen`erator, which `testgen` will invoke to instantiate the test
9191
class. Its output will be a test file containing a point schedule and a
9292
(mechanical) description of the property which needs to pass.
9393

94-
<!-- TODO(sandy): how do we serialize properties? -->
95-
9694
The `runner` tool accepts a test file (as output by `testgen`) and
9795
a *shrink index* (see @sec:shrinking), and spins up simulated peers
9896
corresponding to the embedded point schedule. The `runner` tool will then
@@ -112,24 +110,79 @@ can compare the final state of the NUT (as observed by the testing peer) and
112110
ensure the desired property holds. Depending on the result of the test and the
113111
state of the shrink index, we will perform different actions (see @sec:exit-codes.)
114112

113+
It is important to note that a single invocation of the composition `runner
114+
. testgen` correponds to a single unit test. Users are encouraged to run this
115+
composition in a loop, to gain the usual assurances given by property tests.
116+
117+
If a given test fails, and shrinking is desired, users can rerun `runner` with
118+
the outputted shrink index. The pair `(testfile, shrinkindex)` is the entirety
119+
of the shrink search state. By explicitly threading this state through flags
120+
and stdout, the system is stateless.
121+
122+
123+
```mermaid
124+
---
125+
title: Basic Testing Workflow
126+
---
127+
sequenceDiagram
128+
129+
participant PSG as Test Generator
130+
actor User
131+
participant Runner as Test Runner
132+
participant NUT as Node Under Test
133+
134+
Note over User,PSG: Choose test property
135+
User ->> PSG: testgen generate TESTCLASS
136+
PSG ->> User: file.test
137+
138+
loop for shrinkIndex
139+
140+
User ->> Runner: runner file.test shrinkIndex
141+
Runner ->> User: file.topology
142+
Note right of Runner: Runner spins-up<br/>simulated peers
143+
User ->> NUT: node --topology-file=file.topology
144+
145+
NUT -> Runner: Peer connection
146+
Runner -) NUT: Mock blocks
147+
NUT -) Runner: Diffused messages
148+
Runner -) NUT: ...
149+
Note over Runner,NUT: Simulation ends
150+
151+
Note right of Runner: Runner observes<br/>final state
152+
Note right of Runner: Runner evaluates<br/>test property
153+
154+
alt testBit exit_code KEEP_SHRINKING
155+
Runner ->> User: shrinkIndex
156+
157+
end
158+
159+
end
160+
161+
alt testBit exit_code TEST_FAILED
162+
Runner ->> User: minimal-couterexample
163+
else
164+
Note over Runner,NUT: Test passed
165+
end
166+
```
167+
115168
The `shrinkView` tool accepts a test file and a shrink index, and outputs the
116169
test file corresponding to the given shrink index. This tool is primarily
117170
useful for looking at non-minimal test inputs, eg, when the user doesn't want
118171
to iterate the shrinking all the way down to a minimal example.
119172

120173

121-
### Supported Operations and Flags
174+
#### Supported Operations and Flags
122175

123176
The **test generator** CLI tool supports, at least, the following operations:
124177

125178
- `list-classes` to list all available test classes.
126179
- `generate` to produce a test file for a test class.
127-
This operation has the following optional flags:
180+
This operation has the following optional flags:
128181
- `--seed` to specify a seed for the generator.
129182
- `meta` to access test class metadata, eg the number of `desired-passes`
130183
we expect to run a test for.
131184

132-
On the other hand, the **test runner** CLI tool supports the following optional flags:
185+
The **test runner** CLI tool supports the following optional flags:
133186

134187
- `--shrink-index` to specify the shrunk point schedule to run, according to the
135188
given index. The index values are output by the `runner` itself.
@@ -142,7 +195,7 @@ workflow. For example, users are free to run the entire test suite by looping
142195
over `testgen list-classes`.
143196

144197

145-
### Exit Codes
198+
#### Exit Codes
146199

147200
The `Exit` bit mask enum is used in the following section:
148201

0 commit comments

Comments
 (0)