Need a more civilized test #1922
Description
test on Reactant library hangs at OUT_OF_MEMORY:
(base) dabajabaza@dabajabaza-computer:~$ julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.11.7 (2025-09-08)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(@v1.11) pkg> test Reactant
Testing Reactant
Status `/tmp/jl_Dt8pFz/Project.toml`
[79e6a3ab] Adapt v4.4.0
[4c88cf16] Aqua v0.8.14
[4fba245c] ArrayInterface v7.22.0
[052768ef] CUDA v5.9.5
[f4c16678] DLFP8Types v0.1.0
[31c24e10] Distributions v0.25.122
[7da242da] Enzyme v0.13.108
[7d51a73a] ExplicitImports v1.14.0
[7a1cc6ca] FFTW v1.10.0
[1a297f60] FillArrays v1.15.0
[81dfefd7] Float8s v0.1.1
[587475ba] Flux v0.16.5
[d9f16b24] Functors v0.5.2
[09f84164] HypothesisTests v0.11.6
[63c18a36] KernelAbstractions v0.9.39
[b2108857] Lux v1.27.0
[82251201] LuxLib v1.13.1
[da04e1cc] MPI v0.20.23
[85b6ec6f] MethodAnalysis v1.0.0
[872c559c] NNlib v0.9.31
[6fe1bfb0] OffsetArrays v1.17.0
[0b1bfda6] OneHotArrays v0.2.10
[3bd65402] Optimisers v0.4.6
[21216c6a] Preferences v1.5.0
[6099a3de] PythonCall v0.9.30
[74087812] Random123 v1.7.1
[3c362404] Reactant v0.2.180
[1bc83da4] SafeTestsets v0.1.0
[276daf66] SpecialFunctions v2.6.1
[860ef19b] StableRNGs v1.0.4
[10745b16] Statistics v1.11.1
[2913bbd2] StatsBase v0.34.8
[e88e6eb3] Zygote v0.7.10
[b77e0a4c] InteractiveUtils v1.11.0
[37e2e46d] LinearAlgebra v1.11.0
[9a3f8284] Random v1.11.0
[8dfed614] Test v1.11.0
Status `/tmp/jl_Dt8pFz/Manifest.toml`
[47edcb42] ADTypes v1.20.0
[621f4979] AbstractFFTs v1.5.0
[1520ce14] AbstractTrees v0.4.5
[7d9f7c33] Accessors v0.1.42
[79e6a3ab] Adapt v4.4.0
[66dad0bd] AliasTables v1.1.3
[4c88cf16] Aqua v0.8.14
[dce04be8] ArgCheck v2.5.0
[4fba245c] ArrayInterface v7.22.0
[a9b6321e] Atomix v1.1.2
[ab4f0b2a] BFloat16s v0.6.0
[198e06fe] BangBang v0.4.6
[9718e550] Baselet v0.1.1
[d1d4a3ce] BitFlags v0.1.9
[fa961155] CEnum v0.5.0
[2a0fbf3d] CPUSummary v0.2.7
[052768ef] CUDA v5.9.5
[1af6417a] CUDA_Runtime_Discovery v1.0.0
[082447d4] ChainRules v1.72.6
[d360d2e6] ChainRulesCore v1.26.0
[944b1d66] CodecZlib v0.7.8
[3da002f7] ColorTypes v0.12.1
[5ae59095] Colors v0.13.1
[861a8166] Combinatorics v1.0.3
[38540f10] CommonSolve v0.2.4
[bbf7d656] CommonSubexpressions v0.3.1
[f70d9fcc] CommonWorldInvalidations v1.0.0
[34da2185] Compat v4.18.1
[a33af91c] CompositionsBase v0.1.2
[2569d6c7] ConcreteStructs v0.2.3
[f0e56b4a] ConcurrentUtilities v2.5.0
[992eb4ea] CondaPkg v0.2.33
[187b0558] ConstructionBase v1.6.0
[6add18c4] ContextVariablesX v0.1.3
[adafc99b] CpuId v0.3.1
[a8cc5b0e] Crayons v4.1.1
[f4c16678] DLFP8Types v0.1.0
[9a962f9c] DataAPI v1.16.0
[a93c6f00] DataFrames v1.8.1
[864edb3b] DataStructures v0.19.3
[e2d170a0] DataValueInterfaces v1.0.0
[244e2a9f] DefineSingletons v0.1.2
[8bb1440f] DelimitedFiles v1.9.1
[163ba53b] DiffResults v1.1.0
[b552c78f] DiffRules v1.15.1
[8d63f2c5] DispatchDoctor v0.4.26
[31c24e10] Distributions v0.25.122
[ffbed154] DocStringExtensions v0.9.5
[4e289a0a] EnumX v1.0.5
[7da242da] Enzyme v0.13.108
[f151be2c] EnzymeCore v0.8.17
[460bff9d] ExceptionUnwrapping v0.1.11
[7d51a73a] ExplicitImports v1.14.0
[e2ba6199] ExprTools v0.1.10
[21656369] ExpressionExplorer v1.1.3
[7a1cc6ca] FFTW v1.10.0
[cc61a311] FLoops v0.2.2
[b9860ae5] FLoopsBase v0.1.1
[9aa1b823] FastClosures v0.3.2
[1a297f60] FillArrays v1.15.0
[53c48c17] FixedPointNumbers v0.8.5
[81dfefd7] Float8s v0.1.1
[587475ba] Flux v0.16.5
⌅ [f6369f11] ForwardDiff v1.0.0
[d9f16b24] Functors v0.5.2
[0c68f7d7] GPUArrays v11.3.1
[46192b85] GPUArraysCore v0.2.0
[61eb1bfa] GPUCompiler v1.7.5
[096a3bc2] GPUToolbox v1.0.0
[cd3eb016] HTTP v1.10.19
[076d061b] HashArrayMappedTries v0.2.0
[34004b35] HypergeometricFunctions v0.3.28
[09f84164] HypothesisTests v0.11.6
[7869d1d1] IRTools v0.4.15
[615f187c] IfElse v0.1.1
[22cec73e] InitialValues v0.3.1
[842dd82b] InlineStrings v1.4.5
[3587e190] InverseFunctions v0.1.17
[41ab1584] InvertedIndices v1.3.1
[92d709cd] IrrationalConstants v0.2.6
[82899510] IteratorInterfaceExtensions v1.0.0
[692b3bcd] JLLWrappers v1.7.1
[0f8b85d8] JSON3 v1.14.3
[b14d175d] JuliaVariables v0.2.4
[63c18a36] KernelAbstractions v0.9.39
[929cbde3] LLVM v9.4.4
[8b046642] LLVMLoopInfo v1.0.0
[b964fa9f] LaTeXStrings v1.4.0
[2ab3a3ac] LogExpFunctions v0.3.29
[e6f89c97] LoggingExtras v1.2.0
[b2108857] Lux v1.27.0
[bb33d45b] LuxCore v1.4.2
[82251201] LuxLib v1.13.1
[c2834f40] MLCore v1.0.0
[7e8f7934] MLDataDevices v1.15.1
[d8e11817] MLStyle v0.4.17
[f1d291b0] MLUtils v0.4.8
[da04e1cc] MPI v0.20.23
[3da0fdf6] MPIPreferences v0.1.11
[1914dd2f] MacroTools v0.5.16
[739be429] MbedTLS v1.1.9
[85b6ec6f] MethodAnalysis v1.0.0
[128add7d] MicroCollections v0.2.0
[0b3b1443] MicroMamba v0.1.14
[e1d29d7a] Missings v1.2.0
[872c559c] NNlib v0.9.31
[5da4648a] NVTX v1.0.1
[77ba4419] NaNMath v1.1.3
[71a1bf82] NameResolution v0.1.5
[d8793406] ObjectFile v0.5.0
[6fe1bfb0] OffsetArrays v1.17.0
[0b1bfda6] OneHotArrays v0.2.10
[4d8831e6] OpenSSL v1.6.0
[3bd65402] Optimisers v0.4.6
[bac558e1] OrderedCollections v1.8.1
[90014a1f] PDMats v0.11.36
[69de0a69] Parsers v2.8.3
[fa939f87] Pidfile v1.3.0
[eebad327] PkgVersion v0.3.3
[2dfb63ee] PooledArrays v1.4.3
⌅ [aea7be01] PrecompileTools v1.2.1
[21216c6a] Preferences v1.5.0
[8162dcfd] PrettyPrint v0.2.0
[08abe8d2] PrettyTables v3.1.2
[33c8b6b6] ProgressLogging v0.1.6
[43287f4e] PtrArrays v1.3.0
[6099a3de] PythonCall v0.9.30
[1fd47b50] QuadGK v2.11.2
[74087812] Random123 v1.7.1
[e6cf234a] RandomNumbers v1.6.0
[3c362404] Reactant v0.2.180
[a3311ec8] ReactantCore v0.1.16
[c1ae055f] RealDot v0.1.0
[189a3867] Reexport v1.2.2
[ae029012] Requires v1.3.1
[79098fc4] Rmath v0.9.0
[f2b01f46] Roots v2.2.10
[1bc83da4] SafeTestsets v0.1.0
[431bcebd] SciMLPublic v1.0.0
[7e506255] ScopedValues v1.5.0
[6c6a2e73] Scratch v1.3.0
[91c51154] SentinelArrays v1.4.8
[efcf1570] Setfield v1.1.2
[605ecd9f] ShowCases v0.1.0
[777ac1f9] SimpleBufferStream v1.2.0
[699a6c99] SimpleTraits v0.9.5
[a2af1166] SortingAlgorithms v1.2.2
[dc90abb0] SparseInverseSubset v0.1.2
[276daf66] SpecialFunctions v2.6.1
[171d559e] SplittablesBase v0.1.15
[860ef19b] StableRNGs v1.0.4
[aedffcd0] Static v1.3.1
[90137ffa] StaticArrays v1.9.15
[1e83bf80] StaticArraysCore v1.4.4
[10745b16] Statistics v1.11.1
[82ae8749] StatsAPI v1.7.1
[2913bbd2] StatsBase v0.34.8
[4c63d2b9] StatsFuns v1.5.2
[892a3eda] StringManipulation v0.4.2
[09ab397b] StructArrays v0.7.2
[53d494c1] StructIO v0.3.1
[856f2bd8] StructTypes v1.11.0
[3783bdb8] TableTraits v1.0.1
[bd369af6] Tables v1.12.1
[e689c965] Tracy v0.1.6
[3bb67fe8] TranscodingStreams v0.11.3
[28d57a85] Transducers v0.4.85
[5c2747f8] URIs v1.6.1
[013be700] UnsafeAtomics v0.3.0
[e17b2a0c] UnsafePointers v1.0.0
[d49dbf32] WeightInitializers v1.2.2
[e88e6eb3] Zygote v0.7.10
[700de1a5] ZygoteRules v0.2.7
[d1e2174e] CUDA_Compiler_jll v0.3.0+0
[4ee394cb] CUDA_Driver_jll v13.0.2+0
[76a88914] CUDA_Runtime_jll v0.19.2+0
⌅ [7cc45869] Enzyme_jll v0.0.221+0
[f5851436] FFTW_jll v3.3.11+0
[e33a78d0] Hwloc_jll v2.12.2+0
[1d5cc7b8] IntelOpenMP_jll v2025.2.0+0
[9c1d0b0a] JuliaNVTXCallbacks_jll v0.2.1+0
[dad2f222] LLVMExtra_jll v0.0.38+0
[1d63c593] LLVMOpenMP_jll v18.1.8+0
[ad6e5548] LibTracyClient_jll v0.9.1+6
[94ce4f54] Libiconv_jll v1.18.0+0
[856f044c] MKL_jll v2025.2.0+0
[7cb0a576] MPICH_jll v4.3.2+0
[f1f71cc9] MPItrampoline_jll v5.5.4+0
[9237b28f] MicrosoftMPI_jll v10.1.4+3
[e98f9f5b] NVTX_jll v3.2.2+0
[fe0851c0] OpenMPI_jll v5.0.9+0
[458c3c95] OpenSSL_jll v3.5.4+0
[efe28fd5] OpenSpecFun_jll v0.5.6+0
⌅ [0192cb87] Reactant_jll v0.0.265+0
[f50d1b31] Rmath_jll v0.5.1+0
⌅ [02c8fc9c] XML2_jll v2.13.9+0
[a65dc6b1] Xorg_libpciaccess_jll v0.18.1+0
[1e29f10c] demumble_jll v1.3.0+0
⌅ [f8abcde7] micromamba_jll v1.5.12+0
[1317d2d5] oneTBB_jll v2022.0.0+1
[4d7b5844] pixi_jll v0.41.3+0
[0dad84c5] ArgTools v1.1.2
[56f22d72] Artifacts v1.11.0
[2a0f44e3] Base64 v1.11.0
[ade2ca70] Dates v1.11.0
[8ba89e20] Distributed v1.11.0
[f43a241f] Downloads v1.6.0
[7b1f6079] FileWatching v1.11.0
[9fa8497b] Future v1.11.0
[b77e0a4c] InteractiveUtils v1.11.0
[4af54fe1] LazyArtifacts v1.11.0
[b27032c2] LibCURL v0.6.4
[76f85450] LibGit2 v1.11.0
[8f399da3] Libdl v1.11.0
[37e2e46d] LinearAlgebra v1.11.0
[56ddb016] Logging v1.11.0
[d6f4376e] Markdown v1.11.0
[a63ad114] Mmap v1.11.0
[ca575930] NetworkOptions v1.2.0
[44cfe95a] Pkg v1.11.0
[de0858da] Printf v1.11.0
[3fa0cd96] REPL v1.11.0
[9a3f8284] Random v1.11.0
[ea8e919c] SHA v0.7.0
[9e88b42a] Serialization v1.11.0
[6462fe0b] Sockets v1.11.0
[2f01184e] SparseArrays v1.11.0
[f489334b] StyledStrings v1.11.0
[4607b0f0] SuiteSparse
[fa267f1f] TOML v1.0.3
[a4e569a6] Tar v1.10.0
[8dfed614] Test v1.11.0
[cf7118a7] UUIDs v1.11.0
[4ec0a83e] Unicode v1.11.0
[e66e0078] CompilerSupportLibraries_jll v1.1.1+0
[deac9b47] LibCURL_jll v8.6.0+0
[e37daf67] LibGit2_jll v1.7.2+0
[29816b5a] LibSSH2_jll v1.11.0+1
[c8ffd9c3] MbedTLS_jll v2.28.6+0
[14a3606d] MozillaCACerts_jll v2023.12.12
[4536629a] OpenBLAS_jll v0.3.27+1
[05823500] OpenLibm_jll v0.8.5+0
[bea87d4a] SuiteSparse_jll v7.7.0+0
[83775a58] Zlib_jll v1.2.13+1
[8e850b90] libblastrampoline_jll v5.11.0+0
[8e850ede] nghttp2_jll v1.59.0+0
[3f19e933] p7zip_jll v17.4.0+2
Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading.
┌ Warning: `Adapt.parent_type` is not implemented for Enzyme.TupleArray{Reactant.TracedRNumber{Float64}, (2, 2), 4, 2}. Assuming Enzyme.TupleArray{Reactant.TracedRNumber{Float64}, (2, 2), 4, 2} isn't a wrapped array.
└ @ Reactant ~/.julia/packages/Reactant/zlIsO/src/Reactant.jl:67
Testing Running tests...
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
W0000 00:00:1764475170.318574 215364 cuda_executor.cc:1802] GPU interconnect information not available: INTERNAL: NVML doesn't support extracting fabric info or NVLink is not used by the device.
I0000 00:00:1764475170.318893 215300 service.cc:158] XLA service 0x17671e90 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1764475170.318916 215300 service.cc:166] StreamExecutor device (0): NVIDIA GeForce RTX 5090 Laptop GPU, Compute Capability 12.0a
I0000 00:00:1764475170.319381 215300 se_gpu_pjrt_client.cc:1039] Using BFC allocator.
I0000 00:00:1764475170.319402 215300 gpu_helpers.cc:136] XLA backend allocating 18860310528 bytes on device 0 for BFCAllocator.
I0000 00:00:1764475170.319429 215300 gpu_helpers.cc:177] XLA backend will use up to 6286770176 bytes on device 0 for CollectiveBFCAllocator.
W0000 00:00:1764475170.321061 215300 cuda_executor.cc:1802] GPU interconnect information not available: INTERNAL: NVML doesn't support extracting fabric info or NVLink is not used by the device.
I0000 00:00:1764475170.326314 215300 cuda_dnn.cc:463] Loaded cuDNN version 91400
I0000 00:00:1764475170.358290 215300 cuda_executor.cc:533] failed to allocate 17.56GiB (18860310528 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358372 215300 cuda_executor.cc:533] failed to allocate 15.81GiB (16974278656 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358417 215300 cuda_executor.cc:533] failed to allocate 14.23GiB (15276850176 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358458 215300 cuda_executor.cc:533] failed to allocate 12.80GiB (13749165056 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358501 215300 cuda_executor.cc:533] failed to allocate 11.52GiB (12374248448 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358540 215300 cuda_executor.cc:533] failed to allocate 10.37GiB (11136823296 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358579 215300 cuda_executor.cc:533] failed to allocate 9.33GiB (10023140352 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358616 215300 cuda_executor.cc:533] failed to allocate 8.40GiB (9020825600 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358654 215300 cuda_executor.cc:533] failed to allocate 7.56GiB (8118743040 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358689 215300 cuda_executor.cc:533] failed to allocate 6.80GiB (7306868736 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358723 215300 cuda_executor.cc:533] failed to allocate 6.12GiB (6576181760 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358756 215300 cuda_executor.cc:533] failed to allocate 5.51GiB (5918563328 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358789 215300 cuda_executor.cc:533] failed to allocate 4.96GiB (5326706688 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358821 215300 cuda_executor.cc:533] failed to allocate 4.46GiB (4794035712 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358857 215300 cuda_executor.cc:533] failed to allocate 4.02GiB (4314632192 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358895 215300 cuda_executor.cc:533] failed to allocate 3.62GiB (3883168768 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358928 215300 cuda_executor.cc:533] failed to allocate 3.25GiB (3494851840 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358962 215300 cuda_executor.cc:533] failed to allocate 2.93GiB (3145366528 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.358995 215300 cuda_executor.cc:533] failed to allocate 2.64GiB (2830829824 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359029 215300 cuda_executor.cc:533] failed to allocate 2.37GiB (2547746816 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359061 215300 cuda_executor.cc:533] failed to allocate 2.13GiB (2292972032 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359094 215300 cuda_executor.cc:533] failed to allocate 1.92GiB (2063674880 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359127 215300 cuda_executor.cc:533] failed to allocate 1.73GiB (1857307392 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359158 215300 cuda_executor.cc:533] failed to allocate 1.56GiB (1671576576 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359189 215300 cuda_executor.cc:533] failed to allocate 1.40GiB (1504418816 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359220 215300 cuda_executor.cc:533] failed to allocate 1.26GiB (1353977088 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359250 215300 cuda_executor.cc:533] failed to allocate 1.13GiB (1218579456 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359280 215300 cuda_executor.cc:533] failed to allocate 1.02GiB (1096721664 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359313 215300 cuda_executor.cc:533] failed to allocate 941.32MiB (987049472 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359343 215300 cuda_executor.cc:533] failed to allocate 847.19MiB (888344576 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359373 215300 cuda_executor.cc:533] failed to allocate 762.47MiB (799510272 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359403 215300 cuda_executor.cc:533] failed to allocate 686.22MiB (719559424 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359434 215300 cuda_executor.cc:533] failed to allocate 617.60MiB (647603456 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359463 215300 cuda_executor.cc:533] failed to allocate 555.84MiB (582843136 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359493 215300 cuda_executor.cc:533] failed to allocate 500.26MiB (524558848 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359526 215300 cuda_executor.cc:533] failed to allocate 450.23MiB (472103168 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359556 215300 cuda_executor.cc:533] failed to allocate 405.21MiB (424892928 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359586 215300 cuda_executor.cc:533] failed to allocate 364.69MiB (382403840 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359615 215300 cuda_executor.cc:533] failed to allocate 328.22MiB (344163584 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359644 215300 cuda_executor.cc:533] failed to allocate 295.40MiB (309747456 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359674 215300 cuda_executor.cc:533] failed to allocate 265.86MiB (278772736 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359703 215300 cuda_executor.cc:533] failed to allocate 239.27MiB (250895616 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359733 215300 cuda_executor.cc:533] failed to allocate 215.34MiB (225806080 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359762 215300 cuda_executor.cc:533] failed to allocate 193.81MiB (203225600 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359791 215300 cuda_executor.cc:533] failed to allocate 174.43MiB (182903040 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359819 215300 cuda_executor.cc:533] failed to allocate 156.99MiB (164612864 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359848 215300 cuda_executor.cc:533] failed to allocate 141.29MiB (148151808 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359880 215300 cuda_executor.cc:533] failed to allocate 127.16MiB (133336832 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359909 215300 cuda_executor.cc:533] failed to allocate 114.44MiB (120003328 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359937 215300 cuda_executor.cc:533] failed to allocate 103.00MiB (108003072 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
I0000 00:00:1764475170.359966 215300 cuda_executor.cc:533] failed to allocate 92.70MiB (97202944 bytes) from device: RESOURCE_EXHAUSTED: : CUDA_ERROR_OUT_OF_MEMORY: out of memory
(no further output below for a long time)