Merge branch 'main' into classifier

Author: gperdrizet

.gitattributes

@@ -0,0 +1,3 @@
+# .gitattributes
+
+*.ipynb linguist-vendored

README.md

@@ -1,7 +1,79 @@
-# LLM Detector
-
-Synthetic text detection API in Python using Flask, Celery, Redis, Gunicorn, Nginx and HuggingFace.
+# Malone
 
 ## News
 
-**2024-07-08**: llm_detector is officially part of the Backdrop Build V5 cohort under the tentative name 'Malone' starting today. Check out the [build page](https://backdropbuild.com/builds/v5/cadmus) for updates.
+**2024-07-08**: llm_detector is officially part of the Backdrop Build V5 cohort under the tentative name 'malone' starting today. Check out the backdrop [build page](https://backdropbuild.com/builds/v5/cadmus) for updates.
+
+**2024-07-30**: Malone is live in Beta on Telegram, give it a try [here](https://t.me/the_malone_bot). Note: some Firefox users have reported issues with the botlink, you can also find malone by messaging '*/start*' to @the_malone_bot anywhere you use Telegram.
+
+**2024-08-01**: [Lauch video](https://youtu.be/6zdLcsC9I_I?si=R6knOnxMySDIRKDQ) is up on YouTube. Congrats to all of the other Backdrop Build finishers.
+
+![malone](https://github.com/gperdrizet/llm_detector/blob/main/telegram_bot/assets/malone_A.jpg?raw=true)
+
+Malone is a synthetic text detection service available on [Telegram Messenger](https://telegram.org/), written in Python using [HuggingFace](https://huggingface.co), [scikit-learn](https://scikit-learn.org/stable/), [XGBoost](https://github.com/dmlc/xgboost), [Luigi](https://github.com/spotify/luigi) and [python-telegram-bot](https://github.com/python-telegram-bot/python-telegram-bot), supported by [Flask](https://flask.palletsprojects.com/en/3.0.x), [Celery](https://docs.celeryq.dev/en/stable/index.html), [Redis](https://redis.io/) & [Docker](https://www.docker.com/) and served via [Gunicorn](https://gunicorn.org/) and [Nginx](https://nginx.org/). Malone uses an in-house trained gradient boosting classifier to estimate the probability that a given text was generated by an LLM. It uses a set of engineered features derived from the input text, for more details see the [feature engineering notebooks](https://github.com/gperdrizet/llm_detector/tree/main/classifier/notebooks).
+
+## Table of Contents
+
+1. Features
+2. Where to find malone
+3. Usage
+4. Performance
+5. Demonstration/experimentation notebooks
+6. About the author
+7. Disclaimer
+
+## 1. Features
+
+- **Easily accessible** - use it anywhere you can access Telegram: iOS or Android apps and any web browser.
+- **Simple interface** - no frills, just send the bot text and it will send back the probability that the text was machine generated.
+- **Useful and accurate** - provides a probability that text is synthetic, allowing users to make their own decisions when evaluating content. Maximum likelihood classification accuracy ~90% on held-out test data.
+- **Model agnostic** - malone is not trained to detect the output of a specific LLM, instead, it uses a gradient boosting classifier and a set of numerical features derived from/calibrated on a large corpus of human and synthetic text samples from multiple LLMs.
+- **No logs** - no user data or message contents are ever persisted to disk.
+- **Open source codebase** - malone is an open source project. Clone it, fork it, extend it, modify it, host it yourself and use it the way you want to use it.
+- **Free**
+
+## 2. Where to find malone
+
+Malone is publicly available on Telegram. You can find malone on the [Telegram bot page](https://t.me/the_malone_bot), or just message @the_malone_bot with '/*start*' to start using it.
+
+There are also plans in the works to offer the bare API to interested parties. If that's you, see section 6 below.
+
+## 3. Usage
+
+To use malone you will need a Telegram account. Telegram is free to use and available as an app for iOS and Android. There is also a web version for desktop use.
+
+Once you have a Telegram account, malone is simple to use. Send the bot any 'suspect' text and it will reply with the probability that the text in question was written by a human or generated by an LLM. For smartphone use, a good trick is long press on 'suspect' text and then share it to malone on Telegram via the context menu. Malone is never more that 2 taps away!
+
+![telegram app screenshot](https://github.com/gperdrizet/llm_detector/blob/main/telegram_bot/assets/telegram_screenshot.jpg?raw=true)
+
+Malone can run in two response modes: 'default' and 'verbose'. Default mode returns the probability associated with the most likely class as a percent (e.g. 75% chance a human wrote this). Verbose mode gives a little more detail about the feature values and prediction metrics. Set the mode by messaging '*/set_mode verbose*' or '*/set_mode default*'.
+
+For best results, submitted text must be between 50 and 500 words.
+
+## 4. Performance
+
+Malone is ~90% accurate with a binary log loss of ~0.25 on hold-out test data depending on the model and feature engineering hyperparameters and the specific train/test split (see example confusion matrix below). The miss-classified examples are more or less evenly split between false negatives and false positives.
+
+![XGBoost confusion matrix](https://github.com/gperdrizet/llm_detector/blob/main/classifier/notebooks/figures/XGBoost_confusion_matrix.png?raw=true)
+
+For more details on the classifier training and performance see: [XGBoost experimentation](https://github.com/gperdrizet/llm_detector/blob/main/classifier/notebooks/04.1-XGBoost_classifier_experimentation.ipynb) and [XGBoost finalized](https://github.com/gperdrizet/llm_detector/blob/main/classifier/notebooks/04.2-XGBoost_classifier_finalized.ipynb).
+
+## 5. Demonstration/experimentation notebooks
+
+Most of the testing and benchmarking during the design phase of the project was trialed in Jupyter notebooks before refactoring into modules. These notebooks are the best way to understand the approach and the engineered features used to train the classifier.
+
+1. [Human and synthetic text training data](https://github.com/gperdrizet/llm_detector/blob/main/classifier/notebooks/01-hans_2024_data.ipynb)
+2. [Perplexity ratio score](https://github.com/gperdrizet/llm_detector/blob/main/classifier/notebooks/02.2-perplexity_ratio_score_finalized.ipynb)
+3. [TF-IDF score](https://github.com/gperdrizet/llm_detector/blob/main/classifier/notebooks/03.2-TF-IDF_finalized.ipynb)
+4. [XGBoost classifier](https://github.com/gperdrizet/llm_detector/blob/main/classifier/notebooks/04.2-XGBoost_classifier_finalized.ipynb)
+
+## 6. About the author
+
+My name is Dr. George Perdrizet, I am a biochemistry & molecular biology PhD seeking a career step from academia to professional data science and/or machine learning engineering. This project was conceived from the scientific literature and built solo over the course of a few weeks - I strongly believe that I have a ton to offer the right organization. If you or anyone you know is interested in an ex-researcher from University of Chicago turned builder and data scientist, please reach out, I'd love to learn from and contribute to your project.
+
+- **Email**: <hire.me@perdrizet.org>
+- **LinkedIn**: [linkedin.com/gperdrizet](https://www.linkedin.com/in/gperdrizet/)
+
+## 7. Disclaimer
+
+Malone is an experimental research project meant for educational, informational and entertainment purposes only. Any predictions made are inherently probabilistic in nature and subject to stochastic errors. Text classifications, no matter how high or low the reported probability, should never be interpreted as proof of authorship or the lack thereof in regard to any text submitted for analysis. Decisions about the source or value of any text are made by the user who considers all factors relevant to themselves and their purpose and takes full responsibility for their own judgment any and actions they may take as a result.

api/__main__.py

@@ -1,13 +1,13 @@
 '''Main module to initialize LLMs, set-up and launch Celery & Flask apps
 using either Gunicorn or the Flask development server'''
 
+import pickle
 import api.functions.flask_app as app_funcs
 import api.functions.helper as helper_funcs
 import api.configuration as config
 
 # Start the logger
 logger = helper_funcs.start_logger()
-
 logger.info('Running in %s mode', config.MODE)
 
 if config.MODE == 'testing':
@@ -22,8 +22,34 @@
     reader_model, writer_model = helper_funcs.start_models(logger)
     logger.info('Models started')
 
+    # Load the other scoring assets
+
+    # Load the perplexity ratio Kullback-Leibler kernel density estimate
+    with open(config.PERPLEXITY_RATIO_KLD_KDE, 'rb') as input_file:
+        perplexity_ratio_kld_kde = pickle.load(input_file)
+
+    # Load the TF-IDF luts
+    with open(config.TFIDF_LUT, 'rb') as input_file:
+        tfidf_luts = pickle.load(input_file)
+
+    # Load the TF_IDF Kullback-Leibler kernel density estimate
+    with open(config.TFIDF_SCORE_KLD_KDE, 'rb') as input_file:
+        tfidf_kld_kde = pickle.load(input_file)
+
+    # Load the model
+    with open(config.XGBOOST_CLASSIFIER, 'rb') as input_file:
+        model = pickle.load(input_file)
+
     # Initialize Flask app
-    flask_app = app_funcs.create_flask_celery_app(reader_model, writer_model)
+    flask_app = app_funcs.create_flask_celery_app(
+        reader_model,
+        writer_model,
+        perplexity_ratio_kld_kde,
+        tfidf_luts,
+        tfidf_kld_kde,
+        model
+    )
+
     logger.info('Flask app initialized')
 
 # Start the celery app

api/classes/llm.py

@@ -46,7 +46,7 @@ def __init__(
         self.cpu_cores = cpu_cores
         self.max_new_tokens = max_new_tokens
 
-        # Reserve loading the tokenizer and model for the load method to 
+        # Reserve loading the tokenizer and model for the load method to
         # give the user a chance to override default parameter values
         self.model = None
         self.tokenizer = None
@@ -77,7 +77,7 @@ def load(self) -> None:
         )
 
         # Set the model to evaluation mode to deactivate any dropout
-        # modules the is done to ensure reproducibility of results 
+        # modules the is done to ensure reproducibility of results
         # during evaluation
         self.model.eval()
 

api/configuration.py

@@ -22,7 +22,7 @@
 DATA_PATH=f'{PROJECT_ROOT_PATH}/data'
 
 # Logging stuff
-LOG_LEVEL='DEBUG'
+LOG_LEVEL='INFO'
 LOG_PREFIX='%(levelname)s - %(message)s'
 CLEAR_LOGS=True
 

api/functions/flask_app.py

@@ -2,14 +2,24 @@
 
 from typing import Callable
 import random
-from flask import Flask, request # type: ignore
-from celery import Celery, Task, shared_task # type: ignore
+from flask import Flask, request
+from celery import Celery, Task, shared_task
+from celery.app import trace
 from celery.result import AsyncResult
 from celery.utils.log import get_task_logger
 import api.configuration as config
 import api.functions.scoring as scoring_funcs
 # pylint: disable=W0223
 
+# Comment ##############################################################
+# Code ########################################################################
+
+# Disable return portion task success message log so that
+# user messages don't get logged.
+trace.LOG_SUCCESS = '''\
+Task %(name)s[%(id)s] succeeded in %(runtime)ss\
+'''
+
 def create_celery_app(app: Flask) -> Celery:
     '''Sets up Celery app object'''
 
@@ -41,7 +51,11 @@ def __call__(self, *args: object, **kwargs: object) -> object:
 
 def create_flask_celery_app(
         reader_model: Callable = None,
-        writer_model: Callable = None
+        writer_model: Callable = None,
+        perplexity_ratio_kld_kde: Callable = None,
+        tfidf_luts: Callable = None,
+        tfidf_kld_kde: Callable = None,
+        model: Callable = None
 ) -> Flask:
 
     '''Creates Flask app for use with Celery'''
@@ -67,64 +81,94 @@ def create_flask_celery_app(
     # Get task logger
     logger = get_task_logger(__name__)
 
+
     @shared_task(ignore_result = False)
-    def score_text(suspect_string: str = None, response_mode: str = 'default') -> str:
+    def score_text(
+            suspect_string: str = None,
+            response_mode: str = 'default'
+    ) -> str:
+
         '''Takes a string and scores it, returns a dict.
         containing the author call and the original string'''
 
-        logger.info(f'Submitting for score: {suspect_string}')
-        logger.info(f'Response mode is: {response_mode}')
+        logger.info('Submitting string for score.')
+        logger.info('Response mode is: %s', response_mode)
+
+        # Check to make sure that text is of sane length
+        text_length = len(suspect_string.split(' '))
+
+        if text_length < 50 or text_length > 400:
+
+            reply = '''For best results text should be longer than 50 words and\
+                  shorter than 400 words.'''
+
+        else:
 
-        # Call the real scoring function or mock based on mode
-        if config.MODE == 'testing':
+            # Call the real scoring function or mock based on mode
+            if config.MODE == 'testing':
 
-            # Mock the score with a random float
-            score = [random.uniform(0, 1)]
+                # Mock the score with a random float
+                score = [random.uniform(0, 1)]
 
-            # Threshold the score
-            if score[0] >= 0.5:
-                call = 'human'
+                # Threshold the score
+                if score[0] >= 0.5:
+                    reply = 'Text is human'
 
-            elif score[0] < 0.5:
-                call = 'synthetic'
+                elif score[0] < 0.5:
+                    reply = 'Text is synthetic'
 
-        elif config.MODE == 'production':
+            elif config.MODE == 'production':
 
-            # Call the scoring function
-            response = scoring_funcs.score_string(
-                reader_model,
-                writer_model,
-                suspect_string,
-                response_mode
-            )
+                # Call the scoring function
+                response = scoring_funcs.score_string(
+                    reader_model,
+                    writer_model,
+                    perplexity_ratio_kld_kde,
+                    tfidf_luts,
+                    tfidf_kld_kde,
+                    model,
+                    suspect_string,
+                    response_mode
+                )
 
-            if response_mode == 'default':
+                if response_mode == 'default':
 
-                human_probability = response[0] * 100
-                machine_probability = response[1] * 100
+                    human_probability = response[0] * 100
+                    machine_probability = response[1] * 100
 
-                if human_probability > machine_probability:
-                    reply = f'{human_probability:.1f}% chance that this text was written by a human.'
+                    if human_probability > machine_probability:
+                        reply = f'''{human_probability:.1f}% chance that this text was written by\
+                              a human.'''
 
-                elif human_probability < machine_probability:
-                    reply = f'{machine_probability:.1f}% chance that this text was written by a machine.'
+                    elif human_probability < machine_probability:
+                        reply = f'{machine_probability:.1f}% chance that this text was written by a machine.'
 
-            elif response_mode == 'verbose':
+                elif response_mode == 'verbose':
 
-                features = (f"Fragment length (tokens): {response[2]['Fragment length (tokens)']:.0f}\n"
-                            f"Perplexity: {response[2]['Perplexity']:.2f}\n"
-                            f"Cross-perplexity: {response[2]['Cross-perplexity']:.2f}\n"
-                            f"Perplexity ratio score: {response[2]['Perplexity ratio score']:.3f}\n"
-                            f"Perplexity ratio Kullback-Leibler score: {response[2]['Perplexity ratio Kullback-Leibler score']:.3f}\n"
-                            f"Human TF-IDF: {response[2]['Human TF-IDF']:.2f}\n"
-                            f"Synthetic TF-IDF: {response[2]['Synthetic TF-IDF']:.2f}\n"
-                            f"TF-IDF score: {response[2]['TF-IDF score']:.3f}\n"
-                            f"TF-IDF Kullback-Leibler score: {response[2]['TF-IDF Kullback-Leibler score']:.3f}")
+                    features = ('Fragment length (tokens): '
+                                f"{response[2]['Fragment length (tokens)']:.0f}\n"
+                                'Perplexity: '
+                                f"{response[2]['Perplexity']:.2f}\n"
+                                'Cross-perplexity: '
+                                f"{response[2]['Cross-perplexity']:.2f}\n"
+                                'Perplexity ratio score: '
+                                f"{response[2]['Perplexity ratio score']:.3f}\n"
+                                'Perplexity ratio Kullback-Leibler score: '
+                                f"{response[2]['Perplexity ratio Kullback-Leibler score']:.3f}\n"
+                                'Human TF-IDF: '
+                                f"{response[2]['Human TF-IDF']:.2f}\n"
+                                'Synthetic TF-IDF: '
+                                f"{response[2]['Synthetic TF-IDF']:.2f}\n"
+                                'TF-IDF score: '
+                                f"{response[2]['TF-IDF score']:.3f}\n"
+                                'TF-IDF Kullback-Leibler score: '
+                                f"{response[2]['TF-IDF Kullback-Leibler score']:.3f}")
 
-                reply = f'Class probabilities: human = {response[0]:.3f}, machine = {response[1]:.3f}\n\nFeature values:\n{features}.'
+                    reply = f'''Class probabilities: human = {response[0]:.3f},\
+                          machine = {response[1]:.3f}\n\nFeature values:\n{features}.'''
 
         # Return the result from the output queue
-        return {'author_call': reply, 'text': suspect_string}
+        return {'reply': reply, 'text': suspect_string}
 
     # Set listener for text strings via POST
     @app.post('/submit_text')