Data Scientist/Applied NLP engineer - Dataset and Model Curation

AI/ML

Data Scientist/Applied NLP Engineer - Dataset and Model Curation

Why Polygraf AI?

At Polygraf, we envision a future where AI augments human capabilities without compromising safety, privacy, or ethical standards. Trust in our commitment to building this future with you.

About Polygraf AI

Polygraf AI redefines AI security for critical operations. Our proprietary Small Language Model (SLM) technology enables organizations to detect, explain, and mitigate AI risks – from data leakage and compliance violations to deepfakes and synthetic content – using local, explainable, and auditable AI solutions.

We’re award-winning and investor-backed: named ‘Best in Show’ at SXSW 2025 (winning in Enterprise, Smart Data, FinTech & Future of Work), recognized as Top AI & Data Product (2025), Top AI Governance Product and Top AI Content Detection Product (2024) by Products That Count, and ranked a Top Analytics Startup to Watch Globally by Dealroom.

About the Role

As a Data Scientist / Applied NLP Engineer, you’ll play a central role in improving the quality of our NLP and LLM systems by building the datasets and evaluation processes that drive model performance. You’ll curate and label large-scale text datasets, define annotation policies, benchmark models across releases, analyze failure patterns, and develop automated LLM-powered labeling and QA pipelines. This role combines hands-on NLP, data engineering, and applied machine learning, requiring strong analytical thinking, exceptional attention to detail, and the ability to make sound decisions in ambiguous edge cases. If you enjoy solving complex language problems, improving models through better data, and turning experimentation into scalable production workflows, you’ll thrive on our team.

What You'll Do

Model QA & benchmarking: systematically evaluate entity predictions, identify error patterns, and distinguish model failures from label/data issues; build and maintain evaluation benchmarks, run comparative evaluations across versions, track key metrics, and communicate performance gains and regressions
Dataset curation & labeling: curate labeled datasets at scale: process, filter, audit, and analyze large text corpora and span-labeled data; evaluate candidate sources, design selection strategies for diverse/entity-rich training and eval sets, and create/review/correct entity annotations with careful judgment on ambiguous cases
Annotation systems & data readiness: write and tighten labeling policies (entity definitions, span boundaries, inconsistencies, gaps, edge cases); design and iterate agentic systems for labeling and label fixing; clean, normalize, validate, and convert selected/labeled outputs into training-ready datasets
LLM deployment & automation: wire models into production-like agentic labeling/QA workflows; automate large-scale API runs, retries, batching, and result collection

You Have

Excellent English (IELTS 7.5+ or equivalent) with hands-on dataset labeling and annotation quality control: sample labeling will require deep contextual understanding of text (subtle intent, boundaries, and edge cases), plus clear writing of labeling policies and QA notes
Strong NLP background (ideally sequence labeling/NER/token-level tasks) with experience QA-ing ML models: error analysis, failure clustering, precision/recall trade-offs
Comfort with large-scale text datasets and dataset selection / curation: sampling, filtering, aggregation, scalable human-review workflows, choosing sources, and selecting high-value samples under coverage and diversity constraints
Experience with data post-processing after selection or agentic labeling (validation, cleanup, format conversion, quality gates) and ability to define and revise labeling policies so rules stay consistent and enforceable across edge cases
Experience with model benchmarking (comparative results across checkpoints/systems) and practical prompt engineering for structured extraction / annotation tasks
Experience deploying or operationalizing LLMs in automated pipelines (API integration, batch jobs, retries, throughput/cost awareness) and solid Python for data work and scripting (able to read and extend existing analysis code)
High attention to detail, strong judgment under ambiguity, and clear written communication of decisions and trade-offs

Nice to Have

Multilingual and large-corpus NLP experience: experience working with multilingual or cross-locale text datasets, including language-specific preprocessing, normalization, and the use of embeddings, similarity search, clustering, or retrieval methods to explore, organize, and select high-value samples from large corpora
End-to-end model development experience: hands-on experience training, fine-tuning, evaluating, and iterating on NLP or LLM-based models, with an understanding of how dataset quality, annotation decisions, and model configuration affect downstream performance
Annotation schema and dataset lifecycle management: experience designing large annotation taxonomies, maintaining consistent entity definitions and span-boundary rules as requirements evolve, and managing versioned datasets, experiment tracking, model artifacts, and reproducible release workflows
Production-oriented applied ML judgment: practical experience operating LLM or NLP systems at scale and making informed trade-offs between quality, cost, latency, throughput, and implementation complexity in fast-moving environments with ambiguous requirements and evolving label definitions

As an equal opportunity employer, we highly value diversity and inclusion. We recognize that a diverse team brings unique perspectives, fosters ongoing innovation, and deepens our connection to the global community we serve. If you’re enthusiastic about spearheading the next era of generative AI and leaving a profound impact on how individuals and brands create, we’re eager to have you join our team.

AI/ML