Senior AI Platform Engineer (Python, AWS, Data Pipelines)

Location: remote with the first day onboarding in Warsaw and occasional visits once per quarter

Rate: 170 pln/h on b2b

Project Overview

We are building CaaS (Content as a Service) — a platform that transforms publisher content (PDF textbooks and Excel manifests) into structured, enriched, AI-ready data.

The platform processes content once and exposes it through a unified service layer used by multiple downstream applications.

Key use cases:

• RAG-based Teacher Assistant

• Editorial tooling

• Future AI-powered student-facing products

The goal of this role is to design, build, and maintain a scalable data and AI platform that ingests, processes, enriches, and serves content reliably across multiple environments and consumers.

Responsibilities

Data Engineering & Pipelines

• Build and maintain multi-stage data ingestion pipelines

• Design and implement idempotent, restartable batch processing workflows

• Use S3 as core storage layer for raw and processed data

• Implement pipeline stages including:

  • Content ingestion and book identity assignment

  • PDF-to-markdown conversion (AI OCR)

  • Table of contents and structure extraction

  • Hierarchical chunking

  • Embedding generation

AI / LLM Processing

• Use LLMs and OCR models to extract structured data from PDFs

• Design prompts and context strategies for consistent outputs

• Generate structured metadata and enrich content for downstream use cases

Data Storage & Consistency

• Maintain PostgreSQL (Aurora) as system of record

• Design and maintain SQL schemas and versioned migrations

• Ensure data consistency across:

  • S3

  • PostgreSQL (Aurora)

  • Vector database (Weaviate)

• Implement reconciliation logic across distributed systems

Retrieval & Vector Search

• Work with Weaviate for vector search and semantic retrieval

• Support RAG-based applications

• Design data organization strategies (by subject, country, and client)

APIs & Integration

• Build REST APIs using FastAPI

• Expose content as a service for multiple downstream applications

• Integrate with internal and external systems

Engineering Practices

• Write strongly typed Python code (mypy)

• Follow CI/CD processes with automated checks (ruff, pytest)

• Work across dev / staging / production environments

• Debug distributed data inconsistencies

Key Requirements

Must-have

• Strong Python development experience (production systems)

• AWS experience (S3, Glue, Aurora)

• Experience with data pipelines (ETL / batch processing)

• Strong SQL and PostgreSQL experience

• Experience with schema design and migrations

Nice-to-have

• Experience with LLMs in production (OCR, content processing, enrichment)

• Prompt engineering / context engineering

• Experience with vector databases (Weaviate, Pinecone, Qdrant, pgvector)

• Knowledge of embeddings, semantic search, and RAG

• Experience with FastAPI

• Experience with Airflow / MWAA

• Experience building data platforms serving multiple consumers