Skip to content

Commit 1a82260

Browse files
authored
Use append in example (#122)
1 parent 918ad5b commit 1a82260

File tree

2 files changed

+7
-10
lines changed

2 files changed

+7
-10
lines changed

Project.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
name = "StreamSampling"
22
uuid = "ff63dad9-3335-55d8-95ec-f8139d39e468"
3-
version = "0.7.3"
3+
version = "0.7.4"
44

55
[deps]
66
Accessors = "7d9f7c33-5ae7-4f3b-8dc6-eff91059b697"

docs/src/example.md

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -31,12 +31,10 @@ function generate_file(filename, format)
3131
end
3232
end
3333
elseif format == :arrow
34-
open(Arrow.Writer, filename) do writer
35-
for i in 1:numchunks
36-
starttpl, endtpl = (i-1)*chunktpl+1, min(i*chunktpl, totaltpl)
37-
Arrow.write(writer, (data=map(i -> (a=rand(), b=rand(), c=rand(), d=rand()),
38-
1:endtpl-starttpl+1),))
39-
end
34+
for i in 1:numchunks
35+
starttpl, endtpl = (i-1)*chunktpl+1, min(i*chunktpl, totaltpl)
36+
Arrow.append("random_data.arrow", (data=map(i -> (a=rand(), b=rand(), c=rand(), d=rand()),
37+
1:endtpl-starttpl+1),);file=false)
4038
end
4139
end
4240
end
@@ -129,9 +127,8 @@ rngs = [Xoshiro(i) for i in 1:Threads.nthreads()]
129127

130128
As you can see, the speed-up is not linear in the number of threads for an hdf5 file. This is
131129
mainly due to the fact that accessing the chunks is single-threaded, so one would need to use
132-
`MPI.jl` as explained at https://juliaio.github.io/HDF5.jl/stable/mpi/ to improve the multi-threading
133-
performance. Though, we are already sampling at 500MB/s, which is not bad!
134-
130+
`MPI.jl` as explained at [HDF5.jl/stable/mpi/](https://juliaio.github.io/HDF5.jl/stable/mpi/) to
131+
improve the multi-threading performance. Though, we are already sampling at 500MB/s, which is not bad!
135132
Using `Arrow.jl` gives an even better performance, and a scalability which is better than
136133
linear somehow, reaching a 2GB/s sampling speed!
137134

0 commit comments

Comments
 (0)