Author: David Radley IBM Egeria Maintainer
The Egeria project has made significant strides in the first half of 2023. It introduced a major new version 4.0, along with numerous additional connectors, view services, and a brand-new framework. The community is thriving, actively engaging in webinars, community meetings, face-to-face sessions, and collaborating within the cloud-native workgroup. Below are further details:
Egeria Version 4.0
In April 2023, the Egeria community marked a significant milestone with the release of Egeria Version 4.0. This update brought several notable enhancements and changes. Key highlights include:
- Upgraded Java Level: The release involved an upgrade from Java 11 to a more advanced version, providing improved performance and compatibility.
- Gradle-only Build: Egeria Core transitioned to a gradle-only build system, streamlining the development process and enhancing build efficiency.
- Updated Spring Framework: The release incorporated a later version of the Spring framework, benefiting developers with its enhanced features and functionalities.
- Removal of Deprecated Functions: Deprecated functions were eliminated, ensuring a cleaner and more efficient codebase.
Detailed descriptions of these changes can be found in the release notes.
Notably, Release 4.0 introduced a new feature called Integration Groups to the integration daemon, expanding its capabilities and offering improved integration options.
Performance improvements in Core Egeria
- Over the past few months, significant performance enhancements were implemented in the core libraries of the OMAG Server Platform. These optimizations led to a notable reduction in execution time for various requests, with some experiencing a 50% decrease. Notably, efforts were focused on improving the marshalling and unmarshalling processes for elements within REST APIs and Events.
- One specific change involved reducing the number of fields in the type description of an open metadata instance, such as entities, relationships, and classifications. This modification could impact repository connector implementations that relied on custom instance-type elements.
Core Egeria
In the main branch of Egeria, it is noticeable that the release version is set to 4.1-SNAPSHOT, which indicates that the upcoming release will be version 4.0. The major version number is being updated for three reasons when working in the core Egeria:
- The Java version has been updated from Java 11 to 17 in all builds.
- The maven build is being eliminated.
- There are performance enhancements that may not be fully backward compatible with repository connector implementation.
Moving to Gradle
The Core Egeria repository was transitioned exclusively to using Gradle in all GitHub pipelines. Although the Maven option remained available in the repository, it was planned to be removed once all known consumers of Egeria were prepared.
- The egeria-dev-projects.git repository has also moved to Gradle and Java 17.
- The egeria-samples.git was expected to take place within the following week.
- Regarding the Egeria Development Dojo, the exercises were being adapted to utilize Gradle. This effort was anticipated to be completed prior to the V4.0 release.
Integration Groups
The Integration Daemon can now be dynamically configured while running through a new feature called integration groups. An integration group is a metadata entity that acts as an anchor for a list of integration connector definitions. The integration daemon is configured with a list of integration groups to monitor. At startup, it retrieves the integration group entities from its metadata access server, and any connected integration connector definitions and uses them to configure itself. It continues to monitor the changes to the integration groups in the metadata access server, making adjustments to the connectors it is running as necessary.
Integration Reports
When an integration connector is running, the integration service monitors the changes that the connector is making to open metadata and publishes them in an integration report.
Connectors
- JDBC connector
- The JDBC Connectors were officially released onto Maven Central. The next step was to add them to the open connectors archive so that they are picked up by the OMAG services.
- HMS connector
- The Egeria HMS connector has been enhanced in 5 ways:
- The gradle build has been improved to bring in the latest dependencies. There is now a new optional property that can be set on the gradle build to create an HMS connector that will work with the IBM Cloud® Data Engine service; to do this the property ibmhive in the gradle build; on the command line specify: ./gradlew build -Pibmhive
- Junits have been added so the coverage is now over 70%.
- The HMS connector now correctly extracts columns from HMS external tables that have been created from Apache Spark™ data frames
- Support for HMS v4 has been added
- A related project to the HMS connector is the HMS listener, which runs in the HMS process and issue Egeria Events when tables are added, dropped or altered.
- The Egeria HMS connector has been enhanced in 5 ways:
- The Secrets Store Connector
- The Secrets Store connector is a new type of connector enabling a connector to access its secrets (passwords, certificates, …) securely at runtime. There is an implementation of this new type of connector that works with environment variables. See the connector catalog for more details.
- Apache Atlas Integration Connector
There is a new Apache Atlas integration connector that is able to publish active glossary terms to one or more glossaries in Apache Atlas.
- Server Metadata Security Connector
The Server Metadata Security Connector has been extended with a new optional interface that allows the connector to control authorization requests to glossaries. See the Metadata Security feature for more details.
View services
In this release, the following view services were added to aid organizations in writing user interfaces (UIs):
- Glossary Browser OMVS
Glossary Browser OMVS supports the searching and navigation through a published glossary plus the ability to add feedback (comments, likes and reviews) to its content For further details, please click here.
- Glossary Workflow OMVS
Glossary Workflow OMVS supports the development of glossaries using a controlled workflow process. For further details, please click here.
New User Interface (UI)
The NEW Egeria UI has been shipped as part of the v4 release package.
Cloud-native workgroup
The cloud-native workgroup continues to make progress. The focus is to create runtimes for Kubernetes that support a single server. There is likely to be a specialized runtime for each of the server types. When these servers run, their configuration is provided through the Kubernetes infrastructure rather than a configuration store.
You can find the workgroup updates for Q2 on the Wiki page.
New Framework – The Open Integration Framework (OIF)
During the development of version 4.0, a new framework called the Open Integration Framework (OIF) was introduced. It was designed to provide the interfaces for an integration connector. These interfaces were implemented initially partly in the integration daemon and administration services. The introduction of the OIF has created a cleaner interface for integration connectors.
Changes to the open metadata labs
Two of our open metadata labs have had improvements made:
- Automated Curation includes integration connectors that are configured through integration groups.
- Working with standard models adds examples of creating new glossaries and terms within them, glossary security, various forms of versioning and version history, and the ability to query terms from a single glossary.
Collaboration with Fybrik
Collaboration with Fybric.io open source project on a new connector allowing the platform to plug in Egeria as a Metadata repository. Watch for changes on the egeria-connector GitHub repository if you are interested in more details on this development.
Events
The Egeria community had a busy first half of the year with a face-to-face workshop in Bucharest, focusing on cloud-native deployments and UI consolidation. They also conducted webinars and held community meetings, reviewed releases, and prepared for the annual project review. These events covered various topics, including integration with third-party technologies, new Lineage Types, code quality checks, and more.
Join the Egeria community in their upcoming events to collaborate, contribute, and learn about metadata management and cloud-native operations.
Connecting with the project
We extend our gratitude to the Egeria community for their dedication and contributions. We look forward to the second half of the year, filled with more innovation, collaboration, and growth. Go to our community guide to find out how to find out more about the activities of the Egeria project.
