Skip to content

Major restructuring #135

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 115 commits into
base: main
Choose a base branch
from
Draft

Major restructuring #135

wants to merge 115 commits into from

Conversation

@janheinrichmerker
Copy link
Contributor

@janheinrichmerker janheinrichmerker commented Sep 22, 2025 •
edited
Loading

  • Migrate from elasticsearch-dsl to (type-safe) elasticsearch-pydantic
  • Restructured parsing to use hard-coded (i.e., unit-testable) parsers
  • Migrated monitoring (Flask → FastAPI, now more flexible API)
  • Implement web search result block landing page download (fix Download referenced webpages from search results #6)
  • Remove unused legacy code
  • Updated readme
  • Updated dependencies
  • Updated CI
  • Add dry-run option to test most CLI actions without actual writes
  • Add JSONL export (and local "fake" WARC store for testing)
  • Migrate "legacy" unit tests to work with the current parsing architecture

@janheinrichmerker
Remove legacy CLI
1279799
@janheinrichmerker
Add dependencies
e89f244
@janheinrichmerker
Migrate CLI inputs and outputs to standard library functions
0741359
@janheinrichmerker
Sort dependencies
97a1bab
@janheinrichmerker
Migrate CLI to Cyclopts
a5cd44d
@janheinrichmerker
Migrate config to Cyclopts-CLI-compatible data class
f309818
@janheinrichmerker
Remove Click depenency
4a5a0f6
@janheinrichmerker
Fix required parameter error
cc17706
@janheinrichmerker
Add dry run option
bb3fabb
@janheinrichmerker
Minor fixes
e96b510
@janheinrichmerker
Rename size parameter
4270108
@janheinrichmerker
Clarify/rename type vars
837cd11
@janheinrichmerker
Update dependencies
c8f4791
@janheinrichmerker
Implement Elasticsearch x Pydantic model utils
67950af
@janheinrichmerker
Update ORM to be strictly typed
a2a4fdd
@janheinrichmerker
Migrate code base to ES x Pydantic classes
f327d1d
@janheinrichmerker
Rename/restructure indices/ORMs
c96dfd8
@janheinrichmerker
Rename/restructure indices/ORMs
8e994c3
@janheinrichmerker
Rename/restructure indices/ORMs
41ccbc0
@janheinrichmerker
Rename/restructure CLI
1bca8a7
@janheinrichmerker
Rename/restructure API and importers
44f64d2
@janheinrichmerker
Rename/restructure
5627232
@janheinrichmerker
Rename/restructure web search result block parsers
848c5d1
@janheinrichmerker
Rename/restructure special contents result block parsers
cf3a5ec
@janheinrichmerker
Rename/restructure ORM
37c04bd
@janheinrichmerker
Simplify ES updates
a8b0ff3
@janheinrichmerker
Fix ORM model
5f296b9
@janheinrichmerker
Update result block ORM
f8a13c1
@janheinrichmerker
Add dependencies
ddf666b
@janheinrichmerker
Update readme
349a059
@janheinrichmerker
Implement local export
5f8b0b0
@janheinrichmerker
Fix ORM
8f62b5f
@janheinrichmerker
Remove obsolete CLI
6ed5d14
@janheinrichmerker
Add export CLIs
a61320f
@janheinrichmerker
Decouple parsers
d6e1c8e
@janheinrichmerker
Unify priority types
3d8a058
@janheinrichmerker
Add test utils
79f3fb5
@janheinrichmerker
Add legacy tests conversion
03c9ca9
@janheinrichmerker
Reformat notebook
0e67357
@janheinrichmerker
Add WARC store abstraction layer
e9d6ec6
@janheinrichmerker
Make WARC reading more flexible
250d6bb
@janheinrichmerker
Add test utils and mocked WARC store
ea7f222
@janheinrichmerker
Rename class
1d7dfc1
@janheinrichmerker
Add gitignore
0d513d2
@janheinrichmerker
Add approval test utils
16df6ad
@janheinrichmerker
Fix WARC XML parsing
9bd1751
@janheinrichmerker
Pre-compile XPaths
b9d84da
@janheinrichmerker
Add convenience utils for Memento URLs
ec93d14
@janheinrichmerker
Re-add migrated tests
8eedf71
@janheinrichmerker
Add test data
f3ea32e
@janheinrichmerker
Add export all CLI
0fb0b33
@janheinrichmerker
Fix ORM
67f5851
@janheinrichmerker
Fix version access
f153abc
@janheinrichmerker
Fix parser pattern
4e52384
@janheinrichmerker
Prepare Ray dependencies
d73966b
@janheinrichmerker
Add export commands
ede5dbc
@janheinrichmerker
Update Ray env
a3bef3d
@janheinrichmerker
Add analysis scripts
33a563e
@janheinrichmerker
Update Ray config and commands
44ae1d4
@janheinrichmerker
Fix JSONL export
a227607
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Simrut Simrut Awaiting requested review from Simrut

At least 1 approving review is required to merge this pull request.

Assignees

@janheinrichmerker janheinrichmerker

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Download referenced webpages from search results

2 participants

@janheinrichmerker