• Streams data from Epic SDE, ADT feeds, lab results, and other clinical sources into Databricks for downstream model consumption.
• Curates shared clinical feature tables (patient demographics, labs, vitals, diagnoses, utilization history, imaging metadata) in Databricks/Unity Catalog that multiple AI programs consume for model training, validation, and monitoring.
• Owns RAG Infrastructure, the shared retrieval-augmented generation platform that agentic and generative AI programs use to ground LLM outputs in organizational knowledge.
• Designs and operates document ingestion pipelines: normalizing clinical documents, policies, guidelines, and unstructured data sources into formats ready for embedding and retrieval.
• Implements and optimizes chunking strategies tailored to healthcare content (e.g., preserving clinical note structure, section-aware chunking for guidelines and protocols).
• Manages the embedding pipeline: selecting, tuning, and versioning embedding models (domain-specific clinical models where they outperform general-purpose).
• Administers the vector database: schema design, indexing, metadata management, access controls, and performance tuning.
• Builds and maintains retrieval pipelines: hybrid search (vector + keyword/BM25), reranking, and relevance filtering to maximize retrieval precision for downstream agents and LLM applications.
• Establishes data quality gates for RAG: automated profiling, completeness checks, and accuracy scoring before content enters the vector store.
• Monitors retrieval quality metrics (Precision@K, Recall@K, MRR) and continuously optimize retrieval performance.
• Databricks workspace configuration and Unity Catalog governance.
• Cluster policies, compute management, and cost monitoring.
• Manges user/group management and access control.
• Administrator for Feature Store.
Work is typically performed in an office environment. Accountable for satisfying all job specific obligations and complying with all organization policies and procedures. The specific statements in this profile are not intended to be all-inclusive. They represent typical elements considered necessary to successfully perform the job.
*Relevant experience may be a combination of related work experience and degree obtained (Master’s Degree = 2 years).
Ready to apply? Click below to view the full job posting on the company’s website.