AI Product Manager
About Firmus
Firmus is the leading AI platform for preconstruction design review and risk analysis, helping construction teams identify design-related issues and mitigate financial risk. Recently acquired by The Nemetschek Group through its subsidiary Bluebeam, Firmus is uniquely positioned at the intersection of startup agility and enterprise scale. We now have access to massive resources, an established customer base, and deep industry expertise, while maintaining our fast-moving, innovative startup culture.
Key Responsibilities
Evaluation tooling (product ownership)
- Define and drive the roadmap for the AI agents, and tooling used to author and validate AI models
- Design and own the workflow from development to production.
Model quality & metrics
- Design the evaluation framework for our ML/AI systems, translating raw model outputs into unified business and product metrics.
- Perform statistical analysis and threshold calibration for optimization
- Define success criteria, KPIs, and acceptance thresholds and establish monitoring for models before and after production release
Cross-functional leadership
- Partner daily with SMEs to encode domain expertise into measurable checks
- Partner with AI engineers on evaluation methodology, dataset design, and error analysis
- Communicate model quality and risk to leadership and customers in business terms
Required Qualifications
- 5+ years product management, including 2+ years on ML/AI products
- Familiarity with Large Language Models (LLMs), machine learning concepts, and prompt engineering.
- Hands-on comfort with precision/recall, confusion matrices, threshold tuning, and statistical evaluation methods
- Strong statistical knowledge, experimental design, and problem-solving abilities
- Experience building internal tools or platforms (not just end-user features)
- Ability to translate between domain experts, engineers, and executives
To apply for this position, please submit your resume and cover letter to jobs@firmus.ai
Join the Club
Join our monthly newsletter to get the latest news, tips, and tricks.