Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift without a shared standard for lineage metadata. It requires duplication of effort across pipeline tooling, and deployment of new tools can break existing lineage workflows. Getting useful lineage can seem like a sisyphean task.
Enter OpenLineage, an increasingly adopted open standard for lineage metadata collection. It defines a generic model of run, job, and dataset entities identified using consistent naming strategies. The core lineage model is extensible by defining specific facets to enrich those entities.
Join us at the SF Astronomer offices on September 12th at 6:00 pm PT to learn more about the OpenLineage spec and integrations. You’ll meet other members of the ecosystem, learn about the project’s goals and fundamental design, and participate in a robust discussion about the future of the project.
Speakers and topics TBA.
