Skip to content

Commit a723d33

Browse files
committed
[up]: Documentation update
1 parent ad27066 commit a723d33

File tree

3 files changed

+26
-16
lines changed

3 files changed

+26
-16
lines changed

README.md

Lines changed: 26 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,11 @@
1-
# Generated Text Detection #
1+
<p align="center">
2+
<img src="assets/SA_logo.png" alt="SuperAnnotate Logo" width="100" height="100"/>
3+
</p>
4+
5+
<h1 align="center">
6+
SuperAnnotate <br/>
7+
Generated Text Detection <br/>
8+
</h1>
29

310
[![Version](https://img.shields.io/badge/version-1.0.0-green.svg)]() [![Python 3.11](https://img.shields.io/badge/python-3.11-blue.svg)](https://www.python.org/downloads/release/python-3110/) [![CUDA 12.2](https://img.shields.io/badge/CUDA-12.2-green.svg)](https://developer.nvidia.com/cuda-12-2-0-download-archive)
411

@@ -9,16 +16,21 @@ To integrate the detector with your project on the SuperAnnotate platform, pleas
916

1017
### Model ###
1118

12-
The Generated Text Detection model is based on a fine-tuned RoBERTa Large architecture. Trained on a diverse dataset sourced from multiple open datasets, it excels at classifying text inputs as either generated/synthetic or human-written. \
13-
For more details and access to the model, visit its [Hugging Face Model Hub page](https://huggingface.co/SuperAnnotate/roberta-large-llm-content-detector).
19+
The Generated Text Detection model is built on a fine-tuned RoBERTa Large architecture. It has been extensively trained on a diverse dataset that includes internal generation and subset of RAID train dataset, enabling it to accurately classify text as either generated (synthetic) or human-written. \
20+
This model is optimized for robust detection, offering two configurations based on specific needs:
21+
22+
- **Optimized for Low False Positive Rate (FPR):** [AI Detector](https://huggingface.co/SuperAnnotate/ai-detector)
23+
- **Optimized for High Overall Prediction Accuracy:** [LLM Content Detector V2](https://huggingface.co/SuperAnnotate/roberta-large-llm-content-detector-V2)
24+
25+
For more details and access to the model weights, please refer to the links above on the Hugging Face Model Hub.
1426

1527
## How to run it ##
1628

1729
### API Service Configuration ###
1830

1931
You can deploy the service wherever it is convenient; one of the basic options is on a created EC2 instance. Learn about instance creation and setup [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html). \
2032
Hardware requirements will depend on your on your deployment type. Recommended ec2 instances for deployment type 2:
21-
- **GPU**: [**g3s.xlarge**](https://instances.vantage.sh/aws/ec2/g3s.xlarge)
33+
- **GPU**: [**g4dn.xlarge**](https://instances.vantage.sh/aws/ec2/g4dn.xlarge)
2234
- **CPU**: [**a1.large**](https://instances.vantage.sh/aws/ec2/a1.large)
2335

2436
***NOTES***:
@@ -59,18 +71,16 @@ Hardware requirements will depend on your on your deployment type. Recommended e
5971

6072
### Benchmark ###
6173

62-
The model was evaluated on a benchmark collected from the same datasets used for training, alongside a closed subset of SuperAnnotate. \
63-
However, there are no direct intersections of samples between the training data and the benchmark. \
64-
The benchmark comprises 1k samples, with 200 samples per category. \
65-
The model's performance is compared with open-source solutions and popular API detectors in the table below:
66-
67-
| Model/API | Wikipedia | Reddit QA | SA instruction | Papers | Average |
68-
|--------------------------------------------------------------------------------------------------|----------:|----------:|---------------:|-------:|--------:|
69-
| [Hello-SimpleAI](https://huggingface.co/Hello-SimpleAI/chatgpt-detector-roberta) | **0.97**| 0.95 | 0.82 | 0.69 | 0.86 |
70-
| [RADAR](https://huggingface.co/spaces/TrustSafeAI/RADAR-AI-Text-Detector) | 0.47 | 0.84 | 0.59 | 0.82 | 0.68 |
71-
| [GPTZero](https://gptzero.me) | 0.72 | 0.79 | **0.90**| 0.67 | 0.77 |
72-
| [Originality.ai](https://originality.ai) | 0.91 | **0.97**| 0.77 |**0.93**|**0.89** |
73-
| [LLM content detector](https://huggingface.co/SuperAnnotate/roberta-large-llm-content-detector) | 0.88 | 0.95 | 0.84 | 0.81 | 0.87 |
74+
This solution has been validated using the [RAID](https://raid-bench.xyz/) benchmark, which includes a diverse dataset covering:
75+
- 11 LLM models
76+
- 11 adversarial attacks
77+
- 8 domains
78+
79+
The performance of Binoculars is compared to other detectors on the [RAID leaderboard](https://raid-bench.xyz/leaderboard).
80+
81+
![RAID leaderboard](assets/RAID_leaderboard_oct_2024.png)
82+
83+
This is a snapshot of the leaderboard for October 2024
7484

7585
### Time performance ###
7686

161 KB
Loading

assets/SA_logo.png

445 KB
Loading

0 commit comments

Comments
 (0)