Skip to content

Commit a218742

Browse files
pat-altpat-alt
committed
made appendix independent of EMAR package
1 parent f609dae commit a218742

File tree

11 files changed

+800
-21
lines changed

11 files changed

+800
-21
lines changed

‎appendix.pdf‎

0 Bytes
Binary file not shown.

‎appendix.qmd‎

Lines changed: 59 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,48 @@ cd(projectdir)
2020
## Supplementary Material {#sec-satml-appendices .appendix}
2121

2222
```{julia}
23-
using EMAR
23+
using CSV
24+
using DataFrames
25+
using PrettyTables
26+
2427
www_paper_root = "paper/www/appendix/"
28+
29+
# Helper functions
30+
"""
31+
Helper function to quickly generate a Markdown image include.
32+
"""
33+
function get_img_command(data_names, full_paths, fig_labels; fig_caption="", width=100)
34+
fig_cap = fig_caption == "" ? fig_caption : "$fig_caption "
35+
return [
36+
"![$(fig_cap)Data: $(nm).](/$pth){#$(lbl) width=$(width)%}"
37+
for (nm, pth, lbl) in zip(data_names, full_paths, fig_labels)
38+
]
39+
end
40+
41+
"""
42+
Tabulate the bootstrap results for LaTeX.
43+
"""
44+
function tabulate_bs(df::DataFrame, backend::Val{:latex}, alpha::AbstractFloat=0.05; kwrgs...)
45+
hl = LatexHighlighter(
46+
(data, i, j) -> (j == 5) && data[i, 5] < alpha,
47+
["textbf"]
48+
)
49+
pretty_table(
50+
df;
51+
backend = backend,
52+
highlighters = (hl,),
53+
table_type = :longtable,
54+
longtable_footer = "Continued below.",
55+
header = [
56+
"Metric",
57+
"Data",
58+
"Generator",
59+
"Model",
60+
"p-value",
61+
],
62+
kwrgs...
63+
)
64+
end
2565
```
2666

2767
\FloatBarrier
@@ -54,7 +94,7 @@ println(_str)
5494
str_pattern = "line_chart"
5595
fig_caption = "Evolution of evaluation metrics over the course of the experiment."
5696
full_paths = joinpath.(www_paper_path, readdir(www_paper_path)[contains.(readdir(www_paper_path), str_pattern)])
57-
include_img_commands = EMAR.get_img_command(data_names, full_paths, fig_labels; fig_caption)
97+
include_img_commands = get_img_command(data_names, full_paths, fig_labels; fig_caption)
5898
_str = join(include_img_commands, "\n\n")
5999
println(_str)
60100
```
@@ -80,7 +120,7 @@ println(_str)
80120
str_pattern = "errorbar"
81121
fig_caption = "Evaluation metrics at the end of the experiment."
82122
full_paths = joinpath.(www_paper_path, readdir(www_paper_path)[contains.(readdir(www_paper_path), str_pattern)])
83-
include_img_commands = EMAR.get_img_command(data_names, full_paths, fig_labels; fig_caption)
123+
include_img_commands = get_img_command(data_names, full_paths, fig_labels; fig_caption)
84124
_str = join(include_img_commands, "\n\n")
85125
println(_str)
86126
```
@@ -96,7 +136,8 @@ println(_str)
96136
#| output: asis
97137
#| eval: true
98138
99-
EMAR.aggregate_bs_synthetic(backend=Val(:latex))
139+
df = CSV.read(joinpath(www_paper_path, "bootstrap.csv"), DataFrame)
140+
tabulate_bs(df, Val(:latex))
100141
```
101142

102143
Tests for statistical significance of the estimated MMD metrics. We have highlighted p-values smaller than the significance level $\alpha=0.05$ in bold. Data: Synthetic.
@@ -133,7 +174,7 @@ println(_str)
133174
str_pattern = "line_chart"
134175
fig_caption = "Evolution of evaluation metrics over the course of the experiment."
135176
full_paths = joinpath.(www_paper_path, readdir(www_paper_path)[contains.(readdir(www_paper_path), str_pattern)])
136-
include_img_commands = EMAR.get_img_command(data_names, full_paths, fig_labels; fig_caption)
177+
include_img_commands = get_img_command(data_names, full_paths, fig_labels; fig_caption)
137178
_str = join(include_img_commands, "\n\n")
138179
println(_str)
139180
```
@@ -159,7 +200,7 @@ println(_str)
159200
str_pattern = "errorbar"
160201
fig_caption = "Evaluation metrics at the end of the experiment."
161202
full_paths = joinpath.(www_paper_path, readdir(www_paper_path)[contains.(readdir(www_paper_path), str_pattern)])
162-
include_img_commands = EMAR.get_img_command(data_names, full_paths, fig_labels; fig_caption)
203+
include_img_commands = get_img_command(data_names, full_paths, fig_labels; fig_caption)
163204
_str = join(include_img_commands, "\n\n")
164205
println(_str)
165206
```
@@ -174,7 +215,8 @@ println(_str)
174215
#| output: asis
175216
#| eval: true
176217
177-
EMAR.aggregate_bs_real_world(backend=Val(:latex))
218+
df = CSV.read(joinpath(www_paper_path, "bootstrap.csv"), DataFrame)
219+
tabulate_bs(df, Val(:latex))
178220
```
179221

180222
Tests for statistical significance of the estimated MMD metrics. We have highlighted p-values smaller than the significance level $\alpha=0.05$ in bold. Data: Real-World.
@@ -219,7 +261,7 @@ println(_str)
219261
str_pattern = "line_chart"
220262
fig_caption = "Evolution of evaluation metrics over the course of the experiment."
221263
full_paths = joinpath.(www_paper_path, readdir(www_paper_path)[contains.(readdir(www_paper_path), str_pattern)])
222-
include_img_commands = EMAR.get_img_command(data_names, full_paths, fig_labels; fig_caption)
264+
include_img_commands = get_img_command(data_names, full_paths, fig_labels; fig_caption)
223265
_str = join(include_img_commands, "\n\n")
224266
println(_str)
225267
```
@@ -245,7 +287,7 @@ println(_str)
245287
str_pattern = "errorbar"
246288
fig_caption = "Evaluation metrics at the end of the experiment."
247289
full_paths = joinpath.(www_paper_path, readdir(www_paper_path)[contains.(readdir(www_paper_path), str_pattern)])
248-
include_img_commands = EMAR.get_img_command(data_names, full_paths, fig_labels; fig_caption)
290+
include_img_commands = get_img_command(data_names, full_paths, fig_labels; fig_caption)
249291
_str = join(include_img_commands, "\n\n")
250292
println(_str)
251293
```
@@ -260,7 +302,8 @@ println(_str)
260302
#| output: asis
261303
#| eval: true
262304
263-
EMAR.aggregate_bs_mitigation_synthetic(backend=Val(:latex))
305+
df = CSV.read(joinpath(www_paper_path, "bootstrap_synthetic.csv"), DataFrame)
306+
tabulate_bs(df, Val(:latex))
264307
```
265308

266309
Tests for statistical significance of the estimated MMD metrics using mitigation strategies. We have highlighted p-values smaller than the significance level $\alpha=0.05$ in bold. Data: Synthetic.
@@ -275,7 +318,8 @@ Tests for statistical significance of the estimated MMD metrics using mitigation
275318
#| output: asis
276319
#| eval: true
277320
278-
EMAR.aggregate_bs_mitigation_real_world(backend=Val(:latex))
321+
df = CSV.read(joinpath(www_paper_path, "bootstrap_real_world.csv"), DataFrame)
322+
tabulate_bs(df, Val(:latex))
279323
```
280324

281325
Tests for statistical significance of the estimated MMD metrics using mitigation strategies. We have highlighted p-values smaller than the significance level $\alpha=0.05$ in bold. Data: Real-World.
@@ -311,7 +355,7 @@ println(_str)
311355
str_pattern = "line_chart_latent"
312356
fig_caption = "Evolution of evaluation metrics over the course of the experiment."
313357
full_paths = joinpath.(www_paper_path, readdir(www_paper_path)[contains.(readdir(www_paper_path), str_pattern)])
314-
include_img_commands = EMAR.get_img_command(data_names, full_paths, fig_labels; fig_caption)
358+
include_img_commands = get_img_command(data_names, full_paths, fig_labels; fig_caption)
315359
_str = join(include_img_commands, "\n\n")
316360
println(_str)
317361
```
@@ -337,7 +381,7 @@ println(_str)
337381
str_pattern = "errorbar_chart_latent"
338382
fig_caption = "Evaluation metrics at the end of the experiment."
339383
full_paths = joinpath.(www_paper_path, readdir(www_paper_path)[contains.(readdir(www_paper_path), str_pattern)])
340-
include_img_commands = EMAR.get_img_command(data_names, full_paths, fig_labels; fig_caption)
384+
include_img_commands = get_img_command(data_names, full_paths, fig_labels; fig_caption)
341385
_str = join(include_img_commands, "\n\n")
342386
println(_str)
343387
```
@@ -352,7 +396,8 @@ println(_str)
352396
#| output: asis
353397
#| eval: true
354398
355-
EMAR.aggregate_bs_mitigation_latent(backend=Val(:latex))
399+
df = CSV.read(joinpath(www_paper_path, "bootstrap_latent.csv"), DataFrame)
400+
tabulate_bs(df, Val(:latex))
356401
```
357402

358403
Tests for statistical significance of the estimated MMD metrics using mitigation strategies with Latent Space Search. We have highlighted p-values smaller than the significance level $\alpha=0.05$ in bold.

‎paper/www/appendix/mitigation/bootstrap_latent.csv‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
name,data,generator,model,p_value_mean
2+
MMD,Circles,ClapROAR,Deep Ensemble,1.0
3+
MMD,Circles,ClapROAR,Linear,0.994
4+
MMD,Circles,ClapROAR,MLP,1.0
5+
MMD,Circles,Gravitational,Deep Ensemble,0.998
6+
MMD,Circles,Gravitational,Linear,1.0
7+
MMD,Circles,Gravitational,MLP,1.0
8+
MMD,Circles,Latent (γ=0.5),Deep Ensemble,1.0
9+
MMD,Circles,Latent (γ=0.5),Linear,0.9960000000000001
10+
MMD,Circles,Latent (γ=0.5),MLP,1.0
11+
MMD,Circles,Latent (γ=0.9),Deep Ensemble,1.0
12+
MMD,Circles,Latent (γ=0.9),Linear,0.9960000000000001
13+
MMD,Circles,Latent (γ=0.9),MLP,1.0
14+
MMD,Linearly Separable,ClapROAR,Deep Ensemble,0.33399999999999996
15+
MMD,Linearly Separable,ClapROAR,Linear,0.866
16+
MMD,Linearly Separable,ClapROAR,MLP,0.16799999999999998
17+
MMD,Linearly Separable,Gravitational,Deep Ensemble,0.38
18+
MMD,Linearly Separable,Gravitational,Linear,0.82
19+
MMD,Linearly Separable,Gravitational,MLP,0.0
20+
MMD,Linearly Separable,Latent (γ=0.5),Deep Ensemble,0.0
21+
MMD,Linearly Separable,Latent (γ=0.5),Linear,0.892
22+
MMD,Linearly Separable,Latent (γ=0.5),MLP,0.126
23+
MMD,Linearly Separable,Latent (γ=0.9),Deep Ensemble,0.0
24+
MMD,Linearly Separable,Latent (γ=0.9),Linear,0.8960000000000001
25+
MMD,Linearly Separable,Latent (γ=0.9),MLP,0.0
26+
MMD,Moons,ClapROAR,Deep Ensemble,0.0
27+
MMD,Moons,ClapROAR,Linear,0.0
28+
MMD,Moons,ClapROAR,MLP,0.0
29+
MMD,Moons,Gravitational,Deep Ensemble,0.0
30+
MMD,Moons,Gravitational,Linear,0.0
31+
MMD,Moons,Gravitational,MLP,0.0
32+
MMD,Moons,Latent (γ=0.5),Deep Ensemble,0.0
33+
MMD,Moons,Latent (γ=0.5),Linear,0.0
34+
MMD,Moons,Latent (γ=0.5),MLP,0.0
35+
MMD,Moons,Latent (γ=0.9),Deep Ensemble,0.0
36+
MMD,Moons,Latent (γ=0.9),Linear,0.0
37+
MMD,Moons,Latent (γ=0.9),MLP,0.0
38+
MMD,Overlapping,ClapROAR,Deep Ensemble,0.0
39+
MMD,Overlapping,ClapROAR,Linear,0.0
40+
MMD,Overlapping,ClapROAR,MLP,0.0
41+
MMD,Overlapping,Gravitational,Deep Ensemble,0.0
42+
MMD,Overlapping,Gravitational,Linear,0.0
43+
MMD,Overlapping,Gravitational,MLP,0.0
44+
MMD,Overlapping,Latent (γ=0.5),Deep Ensemble,0.0
45+
MMD,Overlapping,Latent (γ=0.5),Linear,0.0
46+
MMD,Overlapping,Latent (γ=0.5),MLP,0.0
47+
MMD,Overlapping,Latent (γ=0.9),Deep Ensemble,0.0
48+
MMD,Overlapping,Latent (γ=0.9),Linear,0.0
49+
MMD,Overlapping,Latent (γ=0.9),MLP,0.0
50+
PP MMD,Circles,ClapROAR,Deep Ensemble,0.998
51+
PP MMD,Circles,ClapROAR,Linear,0.0
52+
PP MMD,Circles,ClapROAR,MLP,1.0
53+
PP MMD,Circles,Gravitational,Deep Ensemble,1.0
54+
PP MMD,Circles,Gravitational,Linear,0.0
55+
PP MMD,Circles,Gravitational,MLP,1.0
56+
PP MMD,Circles,Latent (γ=0.5),Deep Ensemble,0.998
57+
PP MMD,Circles,Latent (γ=0.5),Linear,0.0
58+
PP MMD,Circles,Latent (γ=0.5),MLP,0.9975
59+
PP MMD,Circles,Latent (γ=0.9),Deep Ensemble,0.998
60+
PP MMD,Circles,Latent (γ=0.9),Linear,0.0
61+
PP MMD,Circles,Latent (γ=0.9),MLP,0.998
62+
PP MMD,Linearly Separable,ClapROAR,Deep Ensemble,0.6980000000000001
63+
PP MMD,Linearly Separable,ClapROAR,Linear,0.094
64+
PP MMD,Linearly Separable,ClapROAR,MLP,0.826
65+
PP MMD,Linearly Separable,Gravitational,Deep Ensemble,0.616
66+
PP MMD,Linearly Separable,Gravitational,Linear,0.096
67+
PP MMD,Linearly Separable,Gravitational,MLP,0.9620000000000001
68+
PP MMD,Linearly Separable,Latent (γ=0.5),Deep Ensemble,0.9480000000000001
69+
PP MMD,Linearly Separable,Latent (γ=0.5),Linear,0.094
70+
PP MMD,Linearly Separable,Latent (γ=0.5),MLP,0.85
71+
PP MMD,Linearly Separable,Latent (γ=0.9),Deep Ensemble,0.9600000000000002
72+
PP MMD,Linearly Separable,Latent (γ=0.9),Linear,0.072
73+
PP MMD,Linearly Separable,Latent (γ=0.9),MLP,0.966
74+
PP MMD,Moons,ClapROAR,Deep Ensemble,0.9620000000000001
75+
PP MMD,Moons,ClapROAR,Linear,0.13399999999999998
76+
PP MMD,Moons,ClapROAR,MLP,0.005
77+
PP MMD,Moons,Gravitational,Deep Ensemble,0.966
78+
PP MMD,Moons,Gravitational,Linear,0.2075
79+
PP MMD,Moons,Gravitational,MLP,0.01
80+
PP MMD,Moons,Latent (γ=0.5),Deep Ensemble,0.9075
81+
PP MMD,Moons,Latent (γ=0.5),Linear,0.0
82+
PP MMD,Moons,Latent (γ=0.5),MLP,0.006
83+
PP MMD,Moons,Latent (γ=0.9),Deep Ensemble,0.93
84+
PP MMD,Moons,Latent (γ=0.9),Linear,0.0275
85+
PP MMD,Moons,Latent (γ=0.9),MLP,0.002
86+
PP MMD,Overlapping,ClapROAR,Deep Ensemble,0.41200000000000003
87+
PP MMD,Overlapping,ClapROAR,Linear,0.13
88+
PP MMD,Overlapping,ClapROAR,MLP,0.33999999999999997
89+
PP MMD,Overlapping,Gravitational,Deep Ensemble,0.544
90+
PP MMD,Overlapping,Gravitational,Linear,0.238
91+
PP MMD,Overlapping,Gravitational,MLP,0.662
92+
PP MMD,Overlapping,Latent (γ=0.5),Deep Ensemble,0.046
93+
PP MMD,Overlapping,Latent (γ=0.5),Linear,0.0
94+
PP MMD,Overlapping,Latent (γ=0.5),MLP,0.07
95+
PP MMD,Overlapping,Latent (γ=0.9),Deep Ensemble,0.196
96+
PP MMD,Overlapping,Latent (γ=0.9),Linear,0.046
97+
PP MMD,Overlapping,Latent (γ=0.9),MLP,0.132
98+
PP MMD (grid),Circles,ClapROAR,Deep Ensemble,0.994
99+
PP MMD (grid),Circles,ClapROAR,Linear,0.0
100+
PP MMD (grid),Circles,ClapROAR,MLP,0.9960000000000001
101+
PP MMD (grid),Circles,Gravitational,Deep Ensemble,0.9920000000000002
102+
PP MMD (grid),Circles,Gravitational,Linear,0.0
103+
PP MMD (grid),Circles,Gravitational,MLP,0.9960000000000001
104+
PP MMD (grid),Circles,Latent (γ=0.5),Deep Ensemble,0.998
105+
PP MMD (grid),Circles,Latent (γ=0.5),Linear,0.0
106+
PP MMD (grid),Circles,Latent (γ=0.5),MLP,0.9924999999999999
107+
PP MMD (grid),Circles,Latent (γ=0.9),Deep Ensemble,0.9940000000000001
108+
PP MMD (grid),Circles,Latent (γ=0.9),Linear,0.0
109+
PP MMD (grid),Circles,Latent (γ=0.9),MLP,0.9880000000000001
110+
PP MMD (grid),Linearly Separable,ClapROAR,Deep Ensemble,0.0
111+
PP MMD (grid),Linearly Separable,ClapROAR,Linear,0.0
112+
PP MMD (grid),Linearly Separable,ClapROAR,MLP,0.0
113+
PP MMD (grid),Linearly Separable,Gravitational,Deep Ensemble,0.0
114+
PP MMD (grid),Linearly Separable,Gravitational,Linear,0.0
115+
PP MMD (grid),Linearly Separable,Gravitational,MLP,0.0
116+
PP MMD (grid),Linearly Separable,Latent (γ=0.5),Deep Ensemble,0.0
117+
PP MMD (grid),Linearly Separable,Latent (γ=0.5),Linear,0.0
118+
PP MMD (grid),Linearly Separable,Latent (γ=0.5),MLP,0.0
119+
PP MMD (grid),Linearly Separable,Latent (γ=0.9),Deep Ensemble,0.0
120+
PP MMD (grid),Linearly Separable,Latent (γ=0.9),Linear,0.0
121+
PP MMD (grid),Linearly Separable,Latent (γ=0.9),MLP,0.044
122+
PP MMD (grid),Moons,ClapROAR,Deep Ensemble,0.128
123+
PP MMD (grid),Moons,ClapROAR,Linear,0.07199999999999998
124+
PP MMD (grid),Moons,ClapROAR,MLP,0.0
125+
PP MMD (grid),Moons,Gravitational,Deep Ensemble,0.2
126+
PP MMD (grid),Moons,Gravitational,Linear,0.1525
127+
PP MMD (grid),Moons,Gravitational,MLP,0.0
128+
PP MMD (grid),Moons,Latent (γ=0.5),Deep Ensemble,0.22
129+
PP MMD (grid),Moons,Latent (γ=0.5),Linear,0.0
130+
PP MMD (grid),Moons,Latent (γ=0.5),MLP,0.0
131+
PP MMD (grid),Moons,Latent (γ=0.9),Deep Ensemble,0.27599999999999997
132+
PP MMD (grid),Moons,Latent (γ=0.9),Linear,0.035
133+
PP MMD (grid),Moons,Latent (γ=0.9),MLP,0.002
134+
PP MMD (grid),Overlapping,ClapROAR,Deep Ensemble,0.296
135+
PP MMD (grid),Overlapping,ClapROAR,Linear,0.19
136+
PP MMD (grid),Overlapping,ClapROAR,MLP,0.374
137+
PP MMD (grid),Overlapping,Gravitational,Deep Ensemble,0.446
138+
PP MMD (grid),Overlapping,Gravitational,Linear,0.32399999999999995
139+
PP MMD (grid),Overlapping,Gravitational,MLP,0.518
140+
PP MMD (grid),Overlapping,Latent (γ=0.5),Deep Ensemble,0.344
141+
PP MMD (grid),Overlapping,Latent (γ=0.5),Linear,0.004
142+
PP MMD (grid),Overlapping,Latent (γ=0.5),MLP,0.49000000000000005
143+
PP MMD (grid),Overlapping,Latent (γ=0.9),Deep Ensemble,0.362
144+
PP MMD (grid),Overlapping,Latent (γ=0.9),Linear,0.052000000000000005
145+
PP MMD (grid),Overlapping,Latent (γ=0.9),MLP,0.41200000000000003

0 commit comments

Comments
 (0)