V1.0.4

- Containerization WSA module.
- Poetry support
- Readme improvements

Author: msorkhpar

README.md

@@ -1,24 +1,38 @@
-# Wiki ES
+# Wiki Entity Summarization Pre-processing
 
 ## Overview
 
-## Getting Started
+This project focuses on the pre-processing steps required for the Wiki Entity Summarization (Wiki ES) project. It
+involves building the necessary databases and loading data from various sources to prepare for the entity summarization
+tasks.
 
-- Build [wikimapper](https://github.com/jcklie/wikimapper) database
+### Server Specifications
+
+For the pre-processing steps, we used an r5a.4xlarge instance on AWS with the following specifications:
+
+vCpu: 16 (AMD EPYC 7571, 16 MiB cache, 2.5 GHz)
+Memory: 128 GB (DDR4, 2667 MT/s)
+Storage: 500 GB (EBS, 2880 Max Bandwidth)
+
+### Getting Started
+
+To get started with the pre-processing, follow these steps:
+
+1. Build the [wikimapper](https://github.com/jcklie/wikimapper) database:
 
 ```shell
 pip install wikimapper
 ````
 
-If you would like to download the latest version run the following
+If you would like to download the latest version, run the following:
 
 ```shell
 EN_WIKI_REDIRECT_AND_PAGES_PATH={your_files_path}
 wikimapper download enwiki-latest --dir $EN_WIKI_REDIRECT_AND_PAGES_PATH
-```   
+```
 
 After having `enwiki-{VERSION}-page.sql.gz`, `enwiki-{VERSION}-redirect.sql.gz`,
-and `enwiki-{VERSION}-page_props.sql.gz` loaded under your data directory:
+and `enwiki-{VERSION}-page_props.sql.gz` loaded under your data directory, run the following commands:
 
 ```shell
 VERSION={VERSION}
@@ -27,8 +41,8 @@ INDEX_DB_PATH="`pwd`/data/index_enwiki-$VERSION.db"
 wikimapper create enwiki-$VERSION --dumpdir $EN_WIKI_REDIRECT_AND_PAGES_PATH --target $INDEX_DB_PATH
 ```
 
-- Now load the created db into our Postgres database,
-  read [pgloader's document](https://pgloader.readthedocs.io/en/latest/install.html) for the installation
+2. Load the created database into the Postgres database:
+   read [pgloader's document](https://pgloader.readthedocs.io/en/latest/install.html) for the installation
 
 ```shell
 ./config-files-generator.sh
@@ -44,7 +58,52 @@ EOT
 pgloader ./sqlite-to-page-migration.load
 ```
 
-After running the experiment we encountered some issues with the `wikimapper` library, so we developed the following script to correct the missing data:
-```shell
+3. Correct missing data: After running the experiments, some issues were encountered with the wikimapper library.
+   To correct the missing data, run the following script:
+
+```shell 
 python3 missing_data_correction.py
-```
+```
+
+## Data Sources
+
+The pre-processing steps involve loading data from the following sources:
+
+- **Wikidata**, [wikidatawiki latest version](https://dumps.wikimedia.org/wikidatawiki/latest/):
+  First, download the latest version of the Wikidata dump. With the dump, you can run the following command to load the
+  metadata of the Wikidata dataset into the Postgres database, and the relationships between the entities into the Neo4j
+  database. This module is called `Wikidata Graph Builder (wdgp)`.
+  ```shell
+  docker-compose up wdgp
+  ```
+- **Wikipedia**, [enwiki lastest version](https://dumps.wikimedia.org/enwiki/latest/):
+  The Wikipedia pages are used to extract the abstract and infobox of the corresponding Wikidata entity. The abstract
+  and infobox are then used to annotate the summary in Wikidata. To provide such information, you need to load the
+  latest version of the Wikipedia dump into the Postgres database. This module is called `Wikipedia Page Extractor (
+  wppe)`.
+    ```shell
+    docker-compose up wppe
+    ```
+
+## Summary Annotation
+
+When both datasets are loaded into the databases, we start processing all the available pages in the Wikipedia dataset
+to extract the abstract and infobox of the corresponding Wikidata entity. Later, these pages are marked from the
+extracted data, and the edges containing the marked pages are marked as candidates. Since Wikidata is a heterogeneous
+graph with multiple types of edges, we need to pick the most relevant edge as a summary between two entities for the
+summarization task. This module is called `Wiki Summary Annotator (wsa)`, and we
+use [DistilBERT](https://arxiv.org/abs/1910.01108)  to filter the most relevant edge.
+
+```shell
+docker-compose up wsa
+```
+
+## Conclusion
+
+By running the above commands, you will have the necessary databases and data loaded to start the Wiki Entity
+Summarization project. The next steps involve providing a set of seed nodes based on your preference along with other
+configuration parameters to get a fully customized Entity Summarization Dataset.
+
+## License
+
+This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.

commons/storage.py

@@ -5,7 +5,6 @@
 from functools import lru_cache
 from pathlib import Path
 from functools import wraps
-from typing import List
 
 from more_itertools import batched
 
@@ -311,11 +310,6 @@ def wrapped(*args, **kwargs):
 
     return wrapped
 
-
-# TODO create indexes if not exists
-# CREATE INDEX summary_summary_for_index FOR ()-[r:SUMMARY]->() ON (r.summary_for);
-# CREATE INDEX wiki_entity_entityName_index FOR (n:WikiEntity) ON (n.entityName);
-
 @lru_cache(maxsize=1024)
 @manage_neo4j_session
 def fetch_relations(subject_qid: str, object_qid: str, session) -> list[tuple[str, str, str]]:

compose.yaml

@@ -6,7 +6,10 @@ services:
     container_name: wikidata_graph_builder
     env_file:
       - configs/wdgb.env
+
     environment:
+      - APP_DUMPFILES_pattern=*pages-articles*xml*.bz2
+      - APP_EXECUTORPOOL_COREPOOLSIZE=20
       - APP_DUMPFILES_DIR=${WIKIDATA_DUMPS_PATH}
       - SPRING_DATASOURCE_URL=jdbc:postgresql://wiki-es-pg:${DB_PORT}/${DB_NAME}
       - SPRING_DATASOURCE_USERNAME=${DB_USER}
@@ -33,6 +36,8 @@ services:
     env_file:
       - configs/wdgb.env
     environment:
+      - APP_DUMPFILES_pattern=*pages-articles*xml*.bz2
+      - APP_EXECUTORPOOL_COREPOOLSIZE=20
       - APP_DUMPFILES_DIR=${WIKIPEDIA_DUMPS_PATH}
       - SPRING_DATASOURCE_URL=jdbc:postgresql://wiki-es-pg:${DB_PORT}/${DB_NAME}
       - SPRING_DATASOURCE_USERNAME=${DB_USER}
@@ -48,6 +53,22 @@ services:
       - wikipedia-dumps:${WIKIPEDIA_DUMPS_PATH}
     networks:
       - wiki-es-network
+  wsa:
+    build:
+      context: .
+      dockerfile: wiki_summary/Dockerfile
+    container_name: wiki_summary_annotator
+    depends_on:
+      neo4j:
+        condition: service_healthy
+      postgres:
+        condition: service_healthy
+    links:
+      - postgres
+    volumes:
+      - ./.env:/app/.env
+    networks:
+      - wiki-es-network
   postgres:
     hostname: wiki-es-pg
     image: 'postgres:latest'

config-files-generator.sh

@@ -7,8 +7,6 @@ POSTGRES_VOLUME_PATH=/data/pg-data
 NEO4J_VOLUME_PATH=/data/neo4j-data
 WIKIDATA_DUMPS_PATH=/data/wikidata/articles/
 WIKIPEDIA_DUMPS_PATH=/data/wikipedia/articlesdump/
-JUPYTER_VOLUME_PATH=/data/jupyter-data
-OUTPUT_VOLUME_PATH=/data/wiki-es-output
 "
 }
 
@@ -20,7 +18,6 @@ DB_USER=wikies
 DB_PASSWORD=password
 NEO4J_USER=neo4j
 NEO4J_PASSWORD=password
-JUPYTER_TOKEN=wikies
 "
 }
 
@@ -35,7 +32,6 @@ NEO4J_HOST=wiki-es-neo
 apps() {
   echo '
 ########## Application ##########
-MAX_DB_CONNECTION_POOL=25
 '
 }
 
@@ -100,6 +96,13 @@ SPRING_DATASOURCE_PASSWORD=\${DB_PASSWORD}
 "
 }
 
+wsa_env() {
+  echo "# Wiki Summarization Annotator
+########################################################################
+MAX_DB_CONNECTION_POOL=25
+"
+}
+
 initialize_env() {
   echo "*************************************"
   local env_file=$1
@@ -128,6 +131,7 @@ initialize_env "./configs/pg.env" pg_env
 initialize_env "./configs/neo4j.env" neo4j_env
 initialize_env "./configs/wdgb.env" wdgb_env
 initialize_env "./configs/wppe.env" wpte_env
+initialize_env "./configs/wsa.env" wsa_env
 
 echo "*************************************"
 echo "Done."