Arena upgrade#7

Open

Chibukach wants to merge 270 commits intomainfrom

This pull request is big! We're only showing the most recent 250 commits

Commits on Jul 14, 2025

remove path
chibu
committed
remove path
chibu
committed
add python path
chibu
committed
move run to scripts
chibu
committed
removed start_gen
chibu
committed
moved pathlib
chibu
committed
update path
chibu
committed
update path
chibu
committed
update path
chibu
committed
move run
chibu
committed
add python path
chibu
committed
update python path
chibu
committed
update path
chibu
committed
add site package to path
chibu
committed
update script path name
chibu
committed
fix config path
chibu
committed
after vllm
chibu
committed

Commits on Jul 15, 2025

clean up
chibu
committed
rename to generate
chibu
committed
reduce questions
chibu
committed
clean up generation
chibu
committed
update config dictionary name
chibu
committed
clean up file paths
chibu
committed
moved based path to top of script
chibu
committed
base judge using answer
chibu
committed
update to judgement
chibu
committed

Commits on Jul 16, 2025

generation to judgement
chibu
committed
missing answer file
chibu
committed
add arenahard yaml
chibu
committed
read from gen judgement
chibu
committed
update to use artifact
chibu
committed
update to reference different task
chibu
committed
debug file location
chibu
committed
add pathlib
chibu
committed
updated answer dir
chibu
committed
update output path for generation
chibu
committed
add answer data
chibu
committed
readd gen
chibu
committed
add pathlib
chibu
committed
update judgment script to use now gen
chibu
committed
clean print
chibu
committed
readd os import
chibu
committed
fix directory definitions
chibu
committed
readd print
chibu
committed
update dir for answer
chibu
committed
test output path
chibu
committed
revert output
chibu
committed
include json dump
chibu
committed

Commits on Jul 17, 2025

revert dump
chibu
committed
change output
chibu
committed
final gen
chibu
committed
update judge to use the generate
chibu
committed
use task id for judgement
chibu
committed
update task to point to judgement model
chibu
committed

Commits on Jul 18, 2025

test with new model
chibu
committed
updated max completion tokens
chibu
committed
test judgement with new model
chibu
committed
if there's a taskid provided to judgement
chibu
committed
fix dict indexing
chibu
committed
reference yaml bench name
chibu
committed
update generate to store based on bench name
chibu
committed
update model name for file output
chibu
committed
update judgement name
chibu
committed
reference the answer model
chibu
committed
add answer example json
chibu
committed
reduced math tokens
chibu
committed

Commits on Jul 22, 2025

update hyperparameters from config
chibu
committed

Commits on Jul 23, 2025

revert to config
chibu
committed
use url
chibu
committed

Commits on Jul 25, 2025

test bucket
chibu
committed

Commits on Jul 28, 2025

update config to use bucket
chibu
committed
pin vllm version
chibu
committed
pin hf hub
chibu
committed

Commits on Jul 29, 2025

pin older vllm version
chibu
committed
update vllm
chibu
committed
vllm so logs show
chibu
committed
lm eval vllm change
chibu
committed
pin transformers version
chibu
committed
hf hub
chibu
committed
pinned transformers and latest vllm
chibu
committed
moved pins
chibu
committed
pin vllm
chibu
committed
revert back to vllm logging to txt file
chibu
committed
revert to main vllm server script
chibu
committed
revert to main vllm server script
chibu
committed

Commits on Jul 31, 2025

Commits on Aug 1, 2025

transformers 4.53 and vllm 0.9.1
chibu
committed
added delay
chibu
committed
remove delay
chibu
committed

Commits on Aug 4, 2025

remove vllm text log
chibu
committed

Commits on Aug 5, 2025

revert to working state
chibu
committed

Commits on Aug 12, 2025

pin triton
chibu
committed

Commits on Aug 14, 2025

update config to match main
chibu
committed
updated libs for judgement
chibu
committed
removed gpu count
chibu
committed

Commits on Aug 18, 2025

updated base task to take pinned library versions
chibu
committed
clean up base task
chibu
committed
update to use previous task
chibu
committed
removed local path line
chibu
committed
use jinja
chibu
committed
update src with template changes
chibu
committed
updated standards path
chibu
committed
update path to same yamls
chibu
committed
update paths
chibu
committed
test path to standards
chibu
committed
update path
chibu
committed
moved file dir save
chibu
committed

Commits on Aug 19, 2025

regenerate yaml
chibu
committed
update api config path
chibu
committed
config gen path
chibu
committed
added questions
chibu
committed
add tmp config files
chibu
committed
changed max token
chibu
committed
reduce tokens
chibu
committed
moved all tmp files
chibu
committed
rename tmp configs
chibu
committed
input max tokens
chibu
committed
clean up
chibu
committed
add parameterised judgement
chibu
committed
import render yaml
chibu
committed
use existing task
chibu
committed
update task parameters
chibu
committed
use current project name
chibu
committed
fix syntax error
chibu
committed
add support for default model as 03
chibu
committed
copy mini 03
chibu
committed
add default queue to pipeline
chibu
committed
remove base pipleine mod
chibu
committed
Merge branch 'main' into arena_upgrade
anmarques
authored

Commits on Aug 20, 2025

full arenahard questions
chibu
committed
updated example workflows
chibu
committed
Merge remote-tracking branch 'refs/remotes/origin/arena_upgrade' into arena_upgrade
chibu
committed
allow use of short questions
chibu
committed
add category support to generate
chibu
committed
added all the model answers
chibu
committed
debug category
chibu
committed
update dict
chibu
committed
retrieve dict value
chibu
committed
changed to quesiton size
chibu
committed
cleanup question size
chibu
committed
update judgement
chibu
committed
remove hardcoded yamls
chibu
committed
added arenahard examples
chibu
committed

Commits on Sep 17, 2025

refactor v2
chibu
committed
reference bench name correctly
chibu
committed
add v0.1 templates
chibu
committed
update judge to refactor v2
chibu
committed
update to use judgement args
chibu
committed
updated examples with v0.1 support
chibu
committed

Commits on Sep 24, 2025

log vllm to file
chibu
committed

Commits on Sep 25, 2025

update tmp_arenahard_file name
chibu
committed

Commits on Sep 26, 2025

add arenahard examples
chibu
committed
add refactoring script
chibu
committed

Commits on Oct 3, 2025

used generic api base
chibu
committed
update target string
chibu
committed

Commits on Oct 6, 2025

fix fstring error
chibu
committed
show logs
chibu
committed
reference judgement
chibu
committed
update the generate task query mechanism
chibu
committed
fix syntax error
chibu
committed
don't start vllm server for gemini model
chibu
committed
add support for api type
chibu
committed
increase parallel
chibu
committed

Commits on Oct 7, 2025

added gemini env var
chibu
committed
test
chibu
committed
comment out log
chibu
committed
run vllm if api key is present
chibu
committed
right model name
chibu
committed
clearer artifact name
chibu
committed
add default answers
chibu
committed
print args
chibu
committed
tmp assertions
chibu
committed
hard code id
chibu
committed
update output path to use judgement
chibu
committed
get judgment results
chibu
committed

Commits on Oct 8, 2025

added api_type
chibu
committed
readd api_type
chibu
committed
add conditional temp
chibu
committed
add conditional temp
chibu
committed
remove asserts
chibu
committed
lower case model
chibu
committed
updated hardcoded task id
chibu
committed
updated hardcoded task id
chibu
committed
tmp move model answer
chibu
committed
fix syntax error
chibu
committed
hardcode answer path
chibu
committed
add make dir
chibu
committed
fix syntax error
chibu
committed
default make dir
chibu
committed
default make dirs
chibu
committed
default makedirs
chibu
committed
change judge name
chibu
committed
reference correct model to judge
chibu
committed
update template arenahards
chibu
committed
rename answer model file saving name
chibu
committed
simplify rename answer model file saving name
chibu
committed
move pull answers
chibu
committed
improve log
chibu
committed
fix order
chibu
committed
retrieve answer_model var
chibu
committed
updated output file name
chibu
committed
artifact object print
chibu
committed
add print
chibu
committed
ensure directory exists before copy
chibu
committed
readd model answer
chibu
committed
full questions
chibu
committed
change output
chibu
committed
output after storage manager pull
chibu
committed
clean up output
chibu
committed
re-add output path
chibu
committed
full questions
chibu
committed
added temperature
chibu
committed
remove hardcode
chibu
committed
get task via taskid
chibu
committed

Commits on Oct 9, 2025

add gpt judgement
chibu
committed

Commits on Oct 10, 2025

Commits on Nov 5, 2025

Commits on Nov 6, 2025

add parent dir
chibu
committed
test with user model
chibu
committed
retry with file move
chibu
committed
support categories
chibu
committed
update v2.0
chibu
committed
latest vllm
chibu
committed
install tags version
chibu
committed
base vllm 11
chibu
committed
latest tagged version
chibu
committed
added reasoning effort to template
chibu
committed
test medium
chibu
committed
test high thinking
chibu
committed
fix at low
chibu
committed

Commits on Nov 7, 2025

test retry
chibu
committed

Commits on Nov 12, 2025

increase wait time
chibu
committed

Commits on Nov 13, 2025

remove sleep
chibu
committed

Commits on Nov 25, 2025