Data derived from the paper GPT detectors are biased against non-native English writers. The study authors carried out a series of experiments passing a number of essays to different GPT detection models. Juxtaposing detector predictions for papers written by native and non-native English writers, the authors argue that GPT detectors disproportionately classify real writing from non-native English writers as AI-generated.
Arguments
- ...
Arguments passed to
pins::pin_read().
Details
A data frame with 6,185 rows and 9 columns:
- kind
Whether the essay was written by a
"Human"or"AI".- .pred_AI
The class probability from the GPT detector that the inputted text was written by AI.
- .pred_class
The uncalibrated class prediction, encoded as
if_else(.pred_AI > .5, "AI", "Human")- detector
The name of the detector used to generate the predictions.
- native
For essays written by humans, whether the essay was written by a native English writer or not. These categorizations are coarse; values of
"Yes"may actually be written by people who do not write with English natively.NAindicates that the text was not written by a human.- name
A label for the experiment that the predictions were generated from.
- model
For essays that were written by AI, the name of the model that generated the essay.
- document_id
A unique identifier for the supplied essay. Some essays were supplied to multiple detectors. Note that some essays are AI-revised derivatives of others.
- prompt
For essays that were written by AI, a descriptor for the form of "prompt engineering" passed to the model.
tibble print
data_detectors()
#> # A tibble: 6,185 x 9
#> kind .pred_AI .pred_class detector native name model document_id prompt
#> <fct> <dbl> <fct> <chr> <chr> <chr> <chr> <dbl> <chr>
#> 1 Human 1.00 AI Sapling No Real~ Human 497 <NA>
#> 2 Human 0.828 AI Crossplag No Real~ Human 278 <NA>
#> 3 Human 0.000214 Human Crossplag Yes Real~ Human 294 <NA>
#> 4 AI 0 Human ZeroGPT <NA> Fake~ GPT3 671 Plain
#> 5 AI 0.00178 Human Originality~ <NA> Fake~ GPT4 717 Eleva~
#> 6 Human 0.000178 Human HFOpenAI Yes Real~ Human 855 <NA>
#> 7 AI 0.992 AI HFOpenAI <NA> Fake~ GPT3 533 Plain
#> 8 AI 0.0226 Human Crossplag <NA> Fake~ GPT4 484 Eleva~
#> 9 Human 0 Human ZeroGPT Yes Real~ Human 781 <NA>
#> 10 Human 1.00 AI Sapling No Real~ Human 460 <NA>
#> # i 6,175 more rowsglimpse()
tibble::glimpse(data_detectors())
#> Rows: 6,185
#> Columns: 9
#> $ kind <fct> Human, Human, Human, AI, AI, Human, AI, AI, Human, Human, ~
#> $ .pred_AI <dbl> 9.999942e-01, 8.281448e-01, 2.137465e-04, 0.000000e+00, 1.~
#> $ .pred_class <fct> AI, AI, Human, Human, Human, Human, AI, Human, Human, AI, ~
#> $ detector <chr> "Sapling", "Crossplag", "Crossplag", "ZeroGPT", "Originali~
#> $ native <chr> "No", "No", "Yes", NA, NA, "Yes", NA, NA, "Yes", "No", NA,~
#> $ name <chr> "Real TOEFL", "Real TOEFL", "Real College Essays", "Fake C~
#> $ model <chr> "Human", "Human", "Human", "GPT3", "GPT4", "Human", "GPT3"~
#> $ document_id <dbl> 497, 278, 294, 671, 717, 855, 533, 484, 781, 460, 591, 11,~
#> $ prompt <chr> NA, NA, NA, "Plain", "Elevate using technical", NA, "Plain~Examples
# \donttest{
data_detectors()
#> # A tibble: 6,185 × 9
#> kind .pred_AI .pred_class detector native name model document_id
#> <fct> <dbl> <fct> <chr> <chr> <chr> <chr> <dbl>
#> 1 Human 1.00 AI Sapling No Real… Human 497
#> 2 Human 0.828 AI Crossplag No Real… Human 278
#> 3 Human 0.000214 Human Crossplag Yes Real… Human 294
#> 4 AI 0 Human ZeroGPT NA Fake… GPT3 671
#> 5 AI 0.00178 Human OriginalityAI NA Fake… GPT4 717
#> 6 Human 0.000178 Human HFOpenAI Yes Real… Human 855
#> 7 AI 0.992 AI HFOpenAI NA Fake… GPT3 533
#> 8 AI 0.0226 Human Crossplag NA Fake… GPT4 484
#> 9 Human 0 Human ZeroGPT Yes Real… Human 781
#> 10 Human 1.00 AI Sapling No Real… Human 460
#> # ℹ 6,175 more rows
#> # ℹ 1 more variable: prompt <chr>
# }