Entity linking scores flip when the API finds the names itself
Summary
WordLift's own resolve tops its entity linking benchmark at 91.8 F1 only when it is handed the names; when the service has to find the names in the text, Google's Cloud Natural Language API scores higher.
Score any shortlist on a gold set of names from your own content, matching on exact Wikidata ID, and read only the benchmark rows that match how you will call the service.
WordLift, which sells schema markup software, has published a head-to-head test of five entity linking services, scored on VoxEL, a public set of news articles annotated in five languages. An answer counted as right only when the Wikidata ID the service returned matched the ID in the gold data. WordLift’s own resolve came first, at 91.8 F1, the single score that combines precision and recall.
The benchmark comes from one of the vendors in it, so the parts worth reading are the rows where resolve loses, and WordLift publishes those. The 91.8 figure belongs to one specific setup: the benchmark hands resolve the exact names to resolve. Give it the article text and make it find the names on its own, the way Google Cloud NL and Dandelion ran, and it scores 87.0, behind Google’s 89.9.
The full table covers 1,005 mentions, with every service on its default thresholds. Google Cloud NL is Google’s Cloud Natural Language API. Dandelion is a hosted API, and Entity-Fishing is an open-source tool you host yourself. The last row is a plain Wikidata search lookup that takes the first hit.
| System | Mentions | F1 | Wrong links |
|---|---|---|---|
| resolve | given | 91.8 | 37 |
| Google Cloud NL | found by the service | 89.9 | 21 |
| resolve | found by the service | 87.0 | 26 |
| Dandelion | found by the service | 85.7 | 40 |
| Entity-Fishing | given | 77.7 | 88 |
| Wikidata search, first hit | name alone | 75.5 | 238 |
What separates the two resolve rows is detection, not disambiguation. Across the full run, 134 mentions came back right from Google and not from resolve, and 125 of those are names resolve never flagged as entities at all. Most are the short second references news prose runs on, “Trump” after “Donald Trump” and “Union” after “European Union”, along with newspaper titles such as Bild and the Times.
So the row to read depends on where the names come from in your own pipeline. If they are already known, from a product feed, an author table, or a CMS field, the given-spans numbers are the relevant ones, and resolve leads in four of the five languages. If the plan is to post raw page text and take back whatever comes out, Google’s row is the fair comparison, and Google also leads in German, 90.1 against 87.0. One difference the F1 scores do not capture: of the services compared, only resolve documents linking to your own identifiers, either from a WordLift graph or up to 5,000 entities sent inline with the request.
The weakest system in the test is the one an agent reaches for by default. A plain Wikidata search call takes a name and returns the top hit with no option to decline, and it produced 238 wrong links against 37 from resolve with the names supplied. For anything that ends up in published markup, a confident wrong QID costs more than a blank. The page still validates, just as a product graph can validate with every price attached to the wrong variant, and the wrong entity fails quietly. Names whose most popular meaning is wrong in context are where that happens: Liverpool the club rather than the city, Java the island rather than the language.
Azure Entity Linking is left out of the scoring because Microsoft is retiring it effective 1 September 2028, after which the feature is no longer supported. Microsoft’s notice points users to Named Entity Recognition or other alternatives. Named Entity Recognition finds and categorises names but does not return the Wikipedia link that Entity Linking did, so anyone moving off Azure still needs one of the other services for the linking step.
What to do
- Work out which half of the table applies before comparing any scores. Supplying your own mention spans and posting raw text are different jobs, and they rank the services differently.
- Build a small gold set from your own content, a hundred or so mentions of the names that matter to you, and score each candidate on exact Wikidata ID match. VoxEL is European news prose, while your ambiguity is more likely to be product names, people named by surname, and brands that share a word with a place.
- Check what each service does when it is unsure, because the defaults differ. Resolve returns an unresolved result with a reason, Dandelion drops anything under its confidence threshold (0.6 by default), and Google Cloud NL leaves the entity unlinked. Prefer a service that can decline over one that always answers.
- Treat WordLift’s second suite, its six development sets, as a catalogue of hard cases rather than a ranking. WordLift says it develops resolve against those sentences, so the scores there favour it.
- If self-hosting is a requirement, Entity-Fishing is the Apache-2.0 option (Java, 3 GB RAM minimum), and it scored lowest of the systems that read context, 77.7 with the names supplied.