# What seven years of vendor audits taught me

By Phoebe Dinh, Founder & CEO of SenBPO · Published August 19, 2026 · Updated September 28, 2026 · 3 min read
Source: https://senbpo.com/insights/auditing-trust-and-safety-vendors/

> The best sign of a good outsourcing partner is not a perfect score. It is what the team does when a number goes in the wrong direction.

For seven years at ByteDance, I reviewed Trust and Safety vendors for TikTok. I read their reports, spoke with moderators and watched how teams handled difficult decisions. Here is what I would check if I were choosing a partner today.

## Key takeaways

- A quality score only means something if you know how it was produced and what changed after it fell.
- Watching a real calibration meeting tells you more than any slide deck.
- Appeals show you where a policy is unclear or where training needs work.
- Staff support and turnover affect quality, so ask how the vendor checks both.
- Bring five specific questions to every vendor review and listen for specific answers.

## Look behind the score

Every vendor can show you a quality score. Ask how they produced it. Which work was reviewed? Were easy cases sampled more often than difficult ones? Can the team show you what changed after a low score?

A report is most useful when it includes the sampling method and a clear explanation of unusual results. A number that never moves may look reassuring, but it tells you very little about the work beneath it.

## Watch a quality meeting

Teams need a way to agree on what a policy means in difficult cases. That happens in calibration meetings, where reviewers and quality leads compare decisions and update the guidance when needed.

Ask to observe one. Do moderators feel comfortable disagreeing? Does the team write down the final decision and use it in training? A meeting that only repeats a scorecard is not enough.

## Read the appeals

When a decision is challenged, a different reviewer should be able to check it. That second look can show where a policy is unclear or where training needs work. Ask how often decisions are overturned and what happens next.

The aim is not to find a perfect overturn rate. The aim is to learn from the cases that reveal a gap.

## Ask about the people doing the work

Trust and Safety teams may review difficult material. Staff support needs to be practical: sensible work rotation, a clear route to help and managers who notice when a team is under strain. Ask how the vendor checks whether those measures work.

Also ask about turnover. When experienced people leave, the team loses product knowledge and has to rebuild it.

## Five questions to bring to your next vendor review

1. How do you choose the work included in a quality score?
2. What changed after the last score that fell?
3. Can we see notes from a recent calibration meeting?
4. Who reviews an appeal, and how are lessons shared?
5. What are you doing to help good team members stay?

Listen for specific answers, not just a slide deck. A strong team can show its process, explain its mistakes and tell you what it has improved.

## Frequently asked questions

### What should I ask a Trust and Safety vendor before signing?

Ask how quality scores are sampled, what changed after the last low score, whether you can observe a calibration meeting, who reviews appeals, and what the vendor does to keep experienced people.

### What is a calibration meeting?

A regular session where reviewers and quality leads compare decisions on difficult cases, agree what the policy means, and write the decision into the guidance and training.

### Is a very high, stable quality score a good sign?

Not on its own. A score that never moves can hide easy sampling. Ask for the sampling method and for examples of what changed after a result fell.

### Why does moderator turnover matter to buyers?

When experienced people leave, the team loses product and policy knowledge. New people need time to rebuild it, and quality can drop in the meantime.

---

_Phoebe Dinh is the founder of SenBPO. Her experience reviewing vendor programs shaped how SenBPO runs quality checks and shares results with clients._

© 2026 SenBPO. All rights reserved. You may quote short parts with credit to SenBPO and a link to https://senbpo.com/insights/auditing-trust-and-safety-vendors/