Yesterday I published the results of my latest testing of German grammar checking systems. Unlike my first round of tests, I included LLMs, in particular ChatGPT, Gemini, and Copilot. I had a nagging feeling that I should include Anthropic's Claude, too, but I didn't find out about Claude until near the end of my testing, and I wanted to be finished, so I decided to worry about Claude later. This was a terrible decision. Later turned out to be only a few hours after I'd published the article, when I unexpectedly found myself with enough free time to play around with the system. Claude proceeded to not only outperform every other system I'd tested, it aced my set of tests with a perfect score.
My test set is hardly exhaustive, but no other system has managed to find and correct all 21 errors in the set. Kudos to Claude and Anthropic!
Here are the updated results of my testing after the addition of Claude to the list:
- Claude: 84 points (100% of possible)
- ChatGPT: 76 (90%)
- LanguageTool: 70 (83%)
- Scribbr/Quillbot: 50 (60%)
- Gemini: 48 (57%)
- Sapling: 41 (49%)
- Copilot: 40 (48%)
- Rechtschreibprüfung24/Korrekturen: 39 (46%)
- Google Docs: 34 (40%)
- GermanCorrector: 31 (37%)
- Online-Spellcheck: 26 (31%)
- Microsoft Word 2010: 16 (19%)
No comments:
Post a Comment