To choose a data annotation provider, first decide what kind of provider you need (a gig marketplace, a large managed workforce, a software platform with labelers attached, or a dedicated team), then test each shortlisted vendor on three things: how they audit quality, how long their annotators stay, and what happens when someone leaves or is banned mid-project. Price per task matters less than most buyers expect. Rework, re-onboarding and inconsistent labels cost more than the rate difference between vendors.
Key takeaways
- Data annotation companies fall into four broad types. Each one is a good fit for some work and a poor fit for other work.
- Ask every vendor how they audit quality, what their annotator retention looks like, and how they replace someone mid-project. Vague answers are a signal.
- The pricing model shapes behavior. Per-task pricing rewards speed; hourly or per-seat pricing rewards consistency.
- Run a paid pilot on your own data before signing a long contract.
- Enterprise teams should also check procurement fit, auditability of the labeling process (the EU AI Act asks for it on high-risk systems) and vendor continuity.
We run dedicated AI training teams for AI companies from Costa Rica, and we have placed more than 300 engineers since 2009. Buyers who talk to us have usually already tried one or two other options. The questions below are the ones we wish more of them had asked before their first contract, including about us.
What data annotation services cover in 2026
A few years ago, data annotation mostly meant drawing boxes around objects or tagging text. That work still exists, but a growing share of what AI companies buy now is judgment work: ranking model responses for RLHF, writing and grading prompts, reviewing code produced by models, checking math and reasoning steps, red teaming, and domain review by people with legal, medical or financial background.
A provider built to label millions of images cheaply is not automatically good at judging whether a model's explanation of a SQL query is correct. Before you compare vendors, write down which of these your project needs:
- Volume labeling: high item counts, short tasks, clear right answers (bounding boxes, classification, transcription).
- Judgment and evaluation: preference ranking, rubric-based grading, prompt writing, where two careful people can disagree.
- Expert review: tasks that need a specific skill (software engineering, a second language, a regulated field).
Most projects are a mix, and the mix tells you which type of provider to talk to.
The four types of data annotation companies
| Provider type | How it works | Good fit | Watch out for |
|---|---|---|---|
| Gig marketplace (Outlier, Remotasks, DataAnnotation and similar) | Large pool of freelancers who pick up tasks through a platform | Early prototypes, bursts of simple volume, testing guidelines | Rotating workers, automated account bans, little control over who does your work |
| Large managed workforce (BPO style) | Vendor employs or contracts big labeling teams, usually priced per task or per hour | Large, stable volume labeling with mature guidelines | Less depth for expert or reasoning tasks, slower to change guidelines |
| Software platform with a workforce attached | You license a labeling tool and the vendor supplies or connects labelers | Teams that want to own the tooling and workflow | You still manage quality yourself; the workforce behind it varies |
| Dedicated or retained team | Named people work only on your project, managed by the vendor | Evaluation, RLHF and expert work that needs consistency over months | Higher hourly rate than the cheapest gig option; needs a minimum commitment |
Gig platforms are useful when you are still figuring out what good labels look like. The problems show up when a project that needs consistency is run on a model designed for turnover. We wrote about that gap in more detail in retained annotation teams vs gig platforms.
Data annotation outsourcing vs building in-house
If your labeling depends on knowledge only your researchers have, or the volume is small and changes every week, keeping it in-house is often faster. Data annotation outsourcing makes sense when the volume is steady, when you need skills you don't want to hire full time, or when your researchers are spending their week labeling instead of doing research. The middle option is a dedicated outsourced team that works inside your tools and process, which is close to how staff augmentation works for software teams.
Three questions to ask any data annotation vendor before signing
1. How do you audit quality?
Every vendor will say they have QA. Ask what it consists of:
- Do they use gold-standard items with known answers, and how often are they mixed into live work?
- What share of completed work is reviewed by a second person, and who reviews the reviewers?
- Do they measure agreement between annotators, and what do they do when it drops?
- Can you see the audit results for your project each week, not a summary at the end?
A good answer names a process and a cadence.
2. What is your annotator retention rate?
Your labeling guidelines only work if the people applying them understand the edge cases. That understanding builds up over weeks. When the workforce rotates, you pay for onboarding again, and the labels drift in ways that are hard to see until they show up in model behavior.
Ask for the number and how they measure it (by person, by project, over what period). If a vendor can't tell you how long the average annotator on a project like yours stays, assume it's short.
3. What happens when a worker is banned or leaves mid-project?
On gig platforms, accounts can be suspended automatically and the person simply stops showing up in your queue. In managed setups, people still resign or get sick. Ask:
- How quickly do you replace someone, and who chooses the replacement?
- How is project knowledge handed over?
- Who pays for the ramp-up time of the new person?
The answer tells you how the vendor will behave in month three, when most annotation projects get hard.
Other things to check before you sign
Security. Ask whether annotators sign NDAs, how access to your data is controlled, where it is stored, which security frameworks the vendor follows and whether it holds any certifications. Check custom requirements before the pilot, not after.
Domain fit. For code review, ask what engineering background the reviewers have. For multilingual work, ask about native speakers versus fluent speakers. For legal or medical review, ask to meet one of the people who would do it.
Tooling and hours. Find out whether the vendor works in your labeling tool or requires theirs; working in yours keeps the data and workflow under your control. And since guidelines change, a team that shares your working hours gets questions answered the same day.
Pricing model. Per-task pricing rewards speed. Hourly or per-seat pricing rewards care and consistency, but you need to know what output to expect per hour. Ask for a quote in the model that fits your work, and ask what is included (QA, project management, replacements).
A paid pilot. Give two or three vendors the same sample of your real data and the same guidelines. Compare accuracy and how they handle ambiguous items. The questions they ask are often more telling than the scores.
What enterprise AI teams should add to the list
Most of this guide is written for AI startups between Series A and Series C, which is where most of our conversations happen. Larger companies building or fine-tuning models have a few extra checks.
Procurement. Enterprise procurement wants a vendor that can sign a master services agreement, pass a security review, and invoice predictably. Gig marketplaces often struggle here because the workforce is not employed or contracted in a way your legal team can review.
Auditability under the EU AI Act. For high-risk AI systems, Article 10 of the EU AI Act asks providers to have data governance practices for training, validation and testing data, and it lists data preparation steps such as annotation and labelling among the things to cover. After the Digital Omnibus changes, the high-risk obligations for most Annex III systems apply from December 2, 2027. If your systems could fall in scope, you will want a vendor who can document who labeled what, under which guidelines, and how it was checked. Your legal team should confirm what applies to you; this is a practical note, not legal advice.
Vendor continuity. Ask what happens to your project if the vendor loses a large client, changes its business model or gets acquired. Rebuilding a team that knows your guidelines takes months.
Where weKnow fits
We are one option, not the right one for every project. If you need millions of simple image labels at the lowest unit price, a large managed workforce will likely suit you better.
Our AI training teams are dedicated: vetted engineers in Latin America who work only on your project, during US hours, in your tools. We handle payroll, compliance, performance reviews and backfills, and the minimum engagement is 20 hours a week per person. The work we are set up for is RLHF evaluation, model output evaluation, prompt quality review, code review and domain review. If you are comparing us with gig platforms specifically, this page lays out the differences.
Frequently asked questions
What are data annotation services?
Data annotation services provide people (and usually tools and QA processes) to label, rank or evaluate data used to train and test AI models. The work ranges from tagging images to grading model responses for RLHF.
How much do data annotation services cost?
It depends on the task, the skill required and the pricing model (per task, per hour or per dedicated seat). Expert and evaluation work costs more than simple labeling. Ask each vendor for a quote on the same sample of work so you can compare.
When should a startup use a gig platform instead of a dedicated team?
Gig platforms work well for early prototypes, one-off bursts and testing guidelines. Once you need the same quality bar week after week, a team that stays usually costs less in rework.
How long does it take to start with a dedicated annotation team?
It varies by vendor and skill. At weKnow, a shortlist and a deployed team typically take under two weeks; specialized domain profiles can take three to four weeks.
Choosing a provider
The right data annotation provider depends on the work. Start by naming what your project actually needs, ask the three questions above, and run a pilot before you commit.
If your project needs a team that stays and learns your guidelines, write to us at yes@weknowinc.com. We'll tell you plainly whether we're a fit.