Skip to content

Commit fa994f6

Browse files
BethanyJepBethanyJep
committed
tweaking distillation hyperparameters, reducing models used.
1 parent 4e9425b commit fa994f6

14 files changed

+322
-123
lines changed

‎.github/chatmodes/study-mode.chatmode.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
---
2+
description: 'Description of the custom chat mode.'
3+
tools: ['changes', 'codebase', 'editFiles', 'extensions', 'fetch', 'findTestFiles', 'githubRepo', 'new', 'openSimpleBrowser', 'problems', 'readCellOutput', 'runCommands', 'runNotebooks', 'runTasks', 'runTests', 'search', 'searchResults', 'terminalLastCommand', 'terminalSelection', 'testFailure', 'usages', 'vscodeAPI', 'microsoft-docs', 'Microsoft Docs', 'Azure MCP', 'azure_azd_up_deploy', 'azure_check_app_status_for_azd_deployment', 'azure_check_pre-deploy', 'azure_check_quota_availability', 'azure_check_region_availability', 'azure_config_deployment_pipeline', 'azure_diagnose_resource', 'azure_generate_azure_cli_command', 'azure_get_auth_state', 'azure_get_available_tenants', 'azure_get_current_tenant', 'azure_get_dotnet_template_tags', 'azure_get_dotnet_templates_for_tag', 'azure_get_selected_subscriptions', 'azure_list_activity_logs', 'azure_open_subscription_picker', 'azure_query_azure_resource_graph', 'azure_recommend_service_config', 'azure_set_current_tenant', 'azure_sign_out_azure_user', 'azure_summarize_topic', 'configurePythonEnvironment', 'getPythonEnvironmentInfo', 'getPythonExecutableCommand', 'installPythonPackage']
4+
---
5+
You are my Adaptive Study Coach inside this codebase.
6+
7+
Goals
8+
- Help me learn {TOPIC} by doing, not just reading.
9+
- Use the repo/workspace context when giving examples or tasks.
10+
- Keep replies concise (≤120 words) unless I say “expand”.
11+
12+
Protocol
13+
1) Kickoff: ask my level (Beginner/Intermediate/Advanced), timebox (e.g., 25m), and one concrete goal.
14+
2) Loop each cycle:
15+
a) 60-sec concept recap tied to the repo.
16+
b) 3 progressive questions (Socratic). Offer HINT 1/2/3 on request.
17+
c) 1 micro-task (≤10 lines of code). Wait for my attempt before revealing.
18+
d) Feedback: what’s right, what to improve, 1 tiny next step.
19+
3) Use file/line references and unified diffs when suggesting edits.
20+
4) Never dump full solutions unless I say “reveal”.
21+
5) After each cycle, give: (i) 3 bullet takeaways, (ii) 2 next micro-tasks.
22+
6) End of session: output 5 spaced-repetition flashcards (Anki cloze format) + a 5-item checklist for tomorrow.
23+
7) Do not update the codebase, your role is to coach me.
24+
25+
Constraints & Style
26+
- One question at a time.
27+
- Prefer examples drawn from this repo or minimal snippets.
28+
- If I seem stuck, switch to “guided mode”: smaller hints, then partial skeletons, then solution.
29+
- Be encouraging but exact; correct misconceptions immediately.
30+
31+
Commands I’ll use
32+
- “hint”, “next”, “reveal”, “harder”, “slower”, “quiz me”, “make flashcards”, “give me a diff”.

‎lab/3-Custom-Politeness-Evaluator/creating-grader.ipynb‎

Lines changed: 3 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -290,7 +290,7 @@
290290
},
291291
{
292292
"cell_type": "code",
293-
"execution_count": 8,
293+
"execution_count": null,
294294
"id": "03eb2456",
295295
"metadata": {},
296296
"outputs": [],
@@ -316,7 +316,7 @@
316316
"Give this a score in the range of 1 to 5 where:\n",
317317
"- 1 means the answer was rude, disrespectful or dismissive\n",
318318
"- 3 means the answer was neutral, neither polite nor rude\n",
319-
"- 5 means the answer had an emoiji followed by a greeting or an acknowledgement of the user question\n",
319+
"- 5 means the answer had an emoji followed by a greeting or an acknowledgement of the user question\n",
320320
"\n",
321321
"### Metric 2: Is the answer helpful?\n",
322322
"\n",
@@ -855,7 +855,7 @@
855855
},
856856
{
857857
"cell_type": "code",
858-
"execution_count": 21,
858+
"execution_count": null,
859859
"id": "bf281e96",
860860
"metadata": {},
861861
"outputs": [
@@ -881,9 +881,7 @@
881881
" \"o3\",\n",
882882
" \"o4-mini\",\n",
883883
" \"gpt-4.1\",\n",
884-
" \"gpt-4.1-mini\",\n",
885884
" \"gpt-4.1-nano\",\n",
886-
" \"gpt-4o\",\n",
887885
" \"gpt-4o-mini\"\n",
888886
"]\n",
889887
"\n",
@@ -1222,16 +1220,6 @@
12221220
" to measure their quality.\n",
12231221
"3. We let Evaluations and Fine Tuning do the rest!"
12241222
]
1225-
},
1226-
{
1227-
"cell_type": "markdown",
1228-
"id": "39763e83",
1229-
"metadata": {},
1230-
"source": [
1231-
"<div style=\"display: flex; align-items: center; justify-content: left; padding: 5px; height: 40px; background: linear-gradient(90deg, #7873f5 0%, #ff6ec4 100%); border-radius: 8px; box-shadow: 0 2px 8px rgba(0,0,0,0.12); font-size: 1.5em; font-weight: bold; color: #fff;\">\n",
1232-
" Next: Be More Precise with RAFT\n",
1233-
"</div>"
1234-
]
12351223
}
12361224
],
12371225
"metadata": {

‎lab/4-Finetuning/basic-fine-tuning.ipynb‎

Lines changed: 59 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -25,10 +25,7 @@
2525
"<div style=\"display: flex; align-items: center; justify-content: left; padding: 5px; height: 40px; background: linear-gradient(90deg, #7873f5 0%, #ff6ec4 100%); border-radius: 8px; box-shadow: 0 2px 8px rgba(0,0,0,0.12); font-size: 1.5em; font-weight: bold; color: #fff;\">\n",
2626
" Step 3: Be More Cost-Effective With Distillation\n",
2727
"</div>\n",
28-
"<br/>\n",
29-
"<div style=\"display: flex; align-items: center; justify-content: left; padding: 5px; height: 40px; background: linear-gradient(90deg, #7873f5 0%, #ff6ec4 100%); border-radius: 8px; box-shadow: 0 2px 8px rgba(0,0,0,0.12); font-size: 1.5em; font-weight: bold; color: #fff;\">\n",
30-
" Step 4: Be More Precise with RAFT\n",
31-
"</div>"
28+
"<br/>"
3229
]
3330
},
3431
{
@@ -280,8 +277,8 @@
280277
"name": "stdout",
281278
"output_type": "stream",
282279
"text": [
283-
"Training file ID: file-47d965f45742443885faad50263a0e8c\n",
284-
"Validation file ID: file-12c4f25aa14941bb9762c0d574ad4133\n"
280+
"Training file ID: file-f0e95ad0245146f986e1cb8e9a82373b\n",
281+
"Validation file ID: file-8b8c2dc873bb43d9aefdfc3b2168fe0c\n"
285282
]
286283
}
287284
],
@@ -314,19 +311,19 @@
314311
},
315312
{
316313
"cell_type": "code",
317-
"execution_count": 8,
314+
"execution_count": 7,
318315
"id": "86ddf91d",
319316
"metadata": {},
320317
"outputs": [
321318
{
322319
"name": "stdout",
323320
"output_type": "stream",
324321
"text": [
325-
"Job ID: ftjob-45902437a97e4bdf8f2333bc1ebdee99\n",
326-
"Status: ftjob-45902437a97e4bdf8f2333bc1ebdee99\n",
322+
"Job ID: ftjob-a66c2404eccb417b9070e9f95a3a96e2\n",
323+
"Status: ftjob-a66c2404eccb417b9070e9f95a3a96e2\n",
327324
"{\n",
328-
" \"id\": \"ftjob-45902437a97e4bdf8f2333bc1ebdee99\",\n",
329-
" \"created_at\": 1758111681,\n",
325+
" \"id\": \"ftjob-a66c2404eccb417b9070e9f95a3a96e2\",\n",
326+
" \"created_at\": 1758295013,\n",
330327
" \"error\": null,\n",
331328
" \"fine_tuned_model\": null,\n",
332329
" \"finished_at\": null,\n",
@@ -342,9 +339,9 @@
342339
" \"seed\": 105,\n",
343340
" \"status\": \"pending\",\n",
344341
" \"trained_tokens\": null,\n",
345-
" \"training_file\": \"file-47d965f45742443885faad50263a0e8c\",\n",
346-
" \"validation_file\": \"file-12c4f25aa14941bb9762c0d574ad4133\",\n",
347-
" \"estimated_finish\": 1758112761,\n",
342+
" \"training_file\": \"file-f0e95ad0245146f986e1cb8e9a82373b\",\n",
343+
" \"validation_file\": \"file-8b8c2dc873bb43d9aefdfc3b2168fe0c\",\n",
344+
" \"estimated_finish\": 1758299439,\n",
348345
" \"integrations\": null,\n",
349346
" \"metadata\": null,\n",
350347
" \"method\": null\n",
@@ -385,17 +382,17 @@
385382
},
386383
{
387384
"cell_type": "code",
388-
"execution_count": 9,
385+
"execution_count": 8,
389386
"id": "f024bb0f",
390387
"metadata": {},
391388
"outputs": [
392389
{
393390
"name": "stdout",
394391
"output_type": "stream",
395392
"text": [
396-
"Fine-tuning job ftjob-45902437a97e4bdf8f2333bc1ebdee99 finished with status: succeeded\n",
393+
"Fine-tuning job ftjob-a66c2404eccb417b9070e9f95a3a96e2 finished with status: succeeded\n",
397394
"Checking other fine-tune jobs for this resource.\n",
398-
"Found 6 fine-tune jobs.\n"
395+
"Found 20 fine-tune jobs.\n"
399396
]
400397
}
401398
],
@@ -443,7 +440,7 @@
443440
},
444441
{
445442
"cell_type": "code",
446-
"execution_count": 10,
443+
"execution_count": 9,
447444
"id": "ebe8fa88",
448445
"metadata": {},
449446
"outputs": [
@@ -454,44 +451,44 @@
454451
"{\n",
455452
" \"data\": [\n",
456453
" {\n",
457-
" \"id\": \"ftevent-b2270bc1c1554cf88360314d6b27cffd\",\n",
458-
" \"created_at\": 1758117577,\n",
454+
" \"id\": \"ftevent-0fcbb3e59a174be0bf017c7aa891db8a\",\n",
455+
" \"created_at\": 1758299162,\n",
459456
" \"level\": \"info\",\n",
460457
" \"message\": \"Training tokens billed: 8000\",\n",
461458
" \"object\": \"fine_tuning.job.event\",\n",
462459
" \"data\": null,\n",
463460
" \"type\": \"message\"\n",
464461
" },\n",
465462
" {\n",
466-
" \"id\": \"ftevent-36487722ede245c38bfd312c7ffba4dc\",\n",
467-
" \"created_at\": 1758117577,\n",
463+
" \"id\": \"ftevent-1daf0625537546ad8fd76f42c31273ba\",\n",
464+
" \"created_at\": 1758299162,\n",
468465
" \"level\": \"info\",\n",
469-
" \"message\": \"Model Evaluation Passed.\",\n",
466+
" \"message\": \"Completed results file: file-0dda62edd87e4ef7bfa1108b19f345ba\",\n",
470467
" \"object\": \"fine_tuning.job.event\",\n",
471468
" \"data\": null,\n",
472469
" \"type\": \"message\"\n",
473470
" },\n",
474471
" {\n",
475-
" \"id\": \"ftevent-ffd856eb590342cb9df5b22e554d6a59\",\n",
476-
" \"created_at\": 1758117577,\n",
472+
" \"id\": \"ftevent-22e8ed38c97440818ac46e7b6e76d15e\",\n",
473+
" \"created_at\": 1758299161,\n",
477474
" \"level\": \"info\",\n",
478-
" \"message\": \"Completed results file: file-876704e7e1ae4c399e8fe08ab856e2b2\",\n",
475+
" \"message\": \"Model Evaluation Passed.\",\n",
479476
" \"object\": \"fine_tuning.job.event\",\n",
480477
" \"data\": null,\n",
481478
" \"type\": \"message\"\n",
482479
" },\n",
483480
" {\n",
484-
" \"id\": \"ftevent-427347ddecbf4755be54acb559284b97\",\n",
485-
" \"created_at\": 1758117551,\n",
481+
" \"id\": \"ftevent-17c825a4e5344dcc8e38fb2679d216c8\",\n",
482+
" \"created_at\": 1758299116,\n",
486483
" \"level\": \"info\",\n",
487484
" \"message\": \"Job succeeded.\",\n",
488485
" \"object\": \"fine_tuning.job.event\",\n",
489486
" \"data\": null,\n",
490487
" \"type\": \"message\"\n",
491488
" },\n",
492489
" {\n",
493-
" \"id\": \"ftevent-808ddf5ee2d72d3808ddf5ee2d72d380\",\n",
494-
" \"created_at\": 1758115747,\n",
490+
" \"id\": \"ftevent-808ddf794d36042808ddf794d3604280\",\n",
491+
" \"created_at\": 1758297273,\n",
495492
" \"level\": \"info\",\n",
496493
" \"message\": \"Step 120: training loss=1.0537992715835571\",\n",
497494
" \"object\": \"fine_tuning.job.event\",\n",
@@ -507,8 +504,8 @@
507504
" \"type\": \"metrics\"\n",
508505
" },\n",
509506
" {\n",
510-
" \"id\": \"ftevent-808ddf5ee277cf2808ddf5ee277cf280\",\n",
511-
" \"created_at\": 1758115737,\n",
507+
" \"id\": \"ftevent-808ddf794cd6a61808ddf794cd6a6180\",\n",
508+
" \"created_at\": 1758297263,\n",
512509
" \"level\": \"info\",\n",
513510
" \"message\": \"Step 110: training loss=0.992326557636261\",\n",
514511
" \"object\": \"fine_tuning.job.event\",\n",
@@ -522,8 +519,8 @@
522519
" \"type\": \"metrics\"\n",
523520
" },\n",
524521
" {\n",
525-
" \"id\": \"ftevent-808ddf5ee218711808ddf5ee21871180\",\n",
526-
" \"created_at\": 1758115727,\n",
522+
" \"id\": \"ftevent-808ddf794c77480808ddf794c7748080\",\n",
523+
" \"created_at\": 1758297253,\n",
527524
" \"level\": \"info\",\n",
528525
" \"message\": \"Step 100: training loss=0.8933090567588806\",\n",
529526
" \"object\": \"fine_tuning.job.event\",\n",
@@ -537,8 +534,8 @@
537534
" \"type\": \"metrics\"\n",
538535
" },\n",
539536
" {\n",
540-
" \"id\": \"ftevent-808ddf5ee1b9130808ddf5ee1b913080\",\n",
541-
" \"created_at\": 1758115717,\n",
537+
" \"id\": \"ftevent-808ddf794c17e9f808ddf794c17e9f80\",\n",
538+
" \"created_at\": 1758297243,\n",
542539
" \"level\": \"info\",\n",
543540
" \"message\": \"Step 90: training loss=1.266113042831421\",\n",
544541
" \"object\": \"fine_tuning.job.event\",\n",
@@ -552,8 +549,8 @@
552549
" \"type\": \"metrics\"\n",
553550
" },\n",
554551
" {\n",
555-
" \"id\": \"ftevent-808ddf5ee159b4f808ddf5ee159b4f80\",\n",
556-
" \"created_at\": 1758115707,\n",
552+
" \"id\": \"ftevent-808ddf794bb88be808ddf794bb88be80\",\n",
553+
" \"created_at\": 1758297233,\n",
557554
" \"level\": \"info\",\n",
558555
" \"message\": \"Step 80: training loss=1.271457314491272\",\n",
559556
" \"object\": \"fine_tuning.job.event\",\n",
@@ -569,8 +566,8 @@
569566
" \"type\": \"metrics\"\n",
570567
" },\n",
571568
" {\n",
572-
" \"id\": \"ftevent-808ddf5ee0fa56e808ddf5ee0fa56e80\",\n",
573-
" \"created_at\": 1758115697,\n",
569+
" \"id\": \"ftevent-808ddf794b592dd808ddf794b592dd80\",\n",
570+
" \"created_at\": 1758297223,\n",
574571
" \"level\": \"info\",\n",
575572
" \"message\": \"Step 70: training loss=1.4114105701446533\",\n",
576573
" \"object\": \"fine_tuning.job.event\",\n",
@@ -613,7 +610,7 @@
613610
},
614611
{
615612
"cell_type": "code",
616-
"execution_count": 11,
613+
"execution_count": 10,
617614
"id": "7088ebaf",
618615
"metadata": {},
619616
"outputs": [
@@ -624,10 +621,10 @@
624621
"{\n",
625622
" \"data\": [\n",
626623
" {\n",
627-
" \"id\": \"ftchkpt-350ee1713844412a9b9ea482995d91df\",\n",
628-
" \"created_at\": 1758116262,\n",
629-
" \"fine_tuned_model_checkpoint\": \"gpt-4.1-2025-04-14.ft-45902437a97e4bdf8f2333bc1ebdee99\",\n",
630-
" \"fine_tuning_job_id\": \"ftjob-45902437a97e4bdf8f2333bc1ebdee99\",\n",
624+
" \"id\": \"ftchkpt-bc8362a9b01b446b9277f91d20b7464f\",\n",
625+
" \"created_at\": 1758297823,\n",
626+
" \"fine_tuned_model_checkpoint\": \"gpt-4.1-2025-04-14.ft-a66c2404eccb417b9070e9f95a3a96e2\",\n",
627+
" \"fine_tuning_job_id\": \"ftjob-a66c2404eccb417b9070e9f95a3a96e2\",\n",
631628
" \"metrics\": {\n",
632629
" \"full_valid_loss\": 1.2260964284301274,\n",
633630
" \"full_valid_mean_token_accuracy\": 0.6798418972332015,\n",
@@ -641,10 +638,10 @@
641638
" \"step_number\": 120\n",
642639
" },\n",
643640
" {\n",
644-
" \"id\": \"ftchkpt-fb1f94a8c9674839965703faea640c27\",\n",
645-
" \"created_at\": 1758116072,\n",
646-
" \"fine_tuned_model_checkpoint\": \"gpt-4.1-2025-04-14.ft-45902437a97e4bdf8f2333bc1ebdee99:ckpt-step-80\",\n",
647-
" \"fine_tuning_job_id\": \"ftjob-45902437a97e4bdf8f2333bc1ebdee99\",\n",
641+
" \"id\": \"ftchkpt-47b40e0ca68b48efa4527c904db9590d\",\n",
642+
" \"created_at\": 1758297635,\n",
643+
" \"fine_tuned_model_checkpoint\": \"gpt-4.1-2025-04-14.ft-a66c2404eccb417b9070e9f95a3a96e2:ckpt-step-80\",\n",
644+
" \"fine_tuning_job_id\": \"ftjob-a66c2404eccb417b9070e9f95a3a96e2\",\n",
648645
" \"metrics\": {\n",
649646
" \"full_valid_loss\": 1.2668834354566492,\n",
650647
" \"full_valid_mean_token_accuracy\": 0.6482213438735178,\n",
@@ -658,10 +655,10 @@
658655
" \"step_number\": 80\n",
659656
" },\n",
660657
" {\n",
661-
" \"id\": \"ftchkpt-94685b61cf314c508538a4aa5a5c9bb7\",\n",
662-
" \"created_at\": 1758115882,\n",
663-
" \"fine_tuned_model_checkpoint\": \"gpt-4.1-2025-04-14.ft-45902437a97e4bdf8f2333bc1ebdee99:ckpt-step-40\",\n",
664-
" \"fine_tuning_job_id\": \"ftjob-45902437a97e4bdf8f2333bc1ebdee99\",\n",
658+
" \"id\": \"ftchkpt-434282d99454403d89b0f410a3c75522\",\n",
659+
" \"created_at\": 1758297443,\n",
660+
" \"fine_tuned_model_checkpoint\": \"gpt-4.1-2025-04-14.ft-a66c2404eccb417b9070e9f95a3a96e2:ckpt-step-40\",\n",
661+
" \"fine_tuning_job_id\": \"ftjob-a66c2404eccb417b9070e9f95a3a96e2\",\n",
665662
" \"metrics\": {\n",
666663
" \"full_valid_loss\": 1.781834651358985,\n",
667664
" \"full_valid_mean_token_accuracy\": 0.5612648221343873,\n",
@@ -704,7 +701,7 @@
704701
},
705702
{
706703
"cell_type": "code",
707-
"execution_count": 12,
704+
"execution_count": 11,
708705
"id": "4a77c5b8",
709706
"metadata": {},
710707
"outputs": [
@@ -713,11 +710,11 @@
713710
"output_type": "stream",
714711
"text": [
715712
"{\n",
716-
" \"id\": \"ftjob-45902437a97e4bdf8f2333bc1ebdee99\",\n",
717-
" \"created_at\": 1758111681,\n",
713+
" \"id\": \"ftjob-a66c2404eccb417b9070e9f95a3a96e2\",\n",
714+
" \"created_at\": 1758295013,\n",
718715
" \"error\": null,\n",
719-
" \"fine_tuned_model\": \"gpt-4.1-2025-04-14.ft-45902437a97e4bdf8f2333bc1ebdee99\",\n",
720-
" \"finished_at\": 1758117577,\n",
716+
" \"fine_tuned_model\": \"gpt-4.1-2025-04-14.ft-a66c2404eccb417b9070e9f95a3a96e2\",\n",
717+
" \"finished_at\": 1758299162,\n",
721718
" \"hyperparameters\": {\n",
722719
" \"batch_size\": 1,\n",
723720
" \"learning_rate_multiplier\": 2.0,\n",
@@ -727,14 +724,14 @@
727724
" \"object\": \"fine_tuning.job\",\n",
728725
" \"organization_id\": null,\n",
729726
" \"result_files\": [\n",
730-
" \"file-876704e7e1ae4c399e8fe08ab856e2b2\"\n",
727+
" \"file-0dda62edd87e4ef7bfa1108b19f345ba\"\n",
731728
" ],\n",
732729
" \"seed\": 105,\n",
733730
" \"status\": \"succeeded\",\n",
734731
" \"trained_tokens\": 10440,\n",
735-
" \"training_file\": \"file-47d965f45742443885faad50263a0e8c\",\n",
736-
" \"validation_file\": \"file-12c4f25aa14941bb9762c0d574ad4133\",\n",
737-
" \"estimated_finish\": 1758115220,\n",
732+
" \"training_file\": \"file-f0e95ad0245146f986e1cb8e9a82373b\",\n",
733+
" \"validation_file\": \"file-8b8c2dc873bb43d9aefdfc3b2168fe0c\",\n",
734+
" \"estimated_finish\": 1758296743,\n",
738735
" \"integrations\": null,\n",
739736
" \"metadata\": null,\n",
740737
" \"method\": null\n",

0 commit comments

Comments
 (0)