MarkLogic Summit November 18, 2009 Betty HarveyElectronic Commerce Connection, Inc. harvey@eccnet.com To maximize success in service oriented development of applications and enterprise wide sharing and reuse initiatives (such as data warehousing), a well-scoped and focused function to manage key metadata artifacts is cost-effective and critical. Gartner Group In response to the growing usage of information technology for conducting business by federal agencies, NARA has made the decision to build the Electronics Records Archive (ERA) system. Its main goal is ?to preserve electronic records independent of the hardware and software that created them? For the ERA system to store, preserve, and provide access to electronic records, it has to cope with the following challenges. Metadata will play a key role in the lifecycle of all records in ERA. Development of the UML Logical Data Model (LDM) metadata is a collaborative effort within NARA.

ERA is the National Archives and Records Administration's strategic initiative to preserve and provide long-term access to uniquely valuable electronic records of the U.S. Government, and to transition government-wide management of the lifecycle of all records into the realm of e-government.http://www.archives.gov/era/ ACEArchival Catalog Entry ARCArchival Research CatalogERAElectronic Records Administration IOCInitial Operating Capability LCDRGLifecycle Data Requirements GuideLDMLogical Data Model NARANational Archive and Records Administration OAISOpen Archival Information System SGMLStandard Generalized Markup Language UMLUnified Modeling Language Multi-year initiative. Initial Operating Capability (IOC) - 6/27/08. Project defined in 5 increments with increased functionality. Currently in increment 3. The IOC had limited functionality and only 5 agencies used the system. Some of the intiial functionality was deferred in order to get to IOC. Goal of 30 agencies using ERA by the end of 2010. 2012 all agencies should be using ERA. Manage the lifecycle of records Provide long term preservation Provide access to digital objects Manage business forms used by NARA and Federal Agencies Records Schedule Transfer Request and Transfer Plan Legal Transfer Instrument Other lesser used forms The "Preservation Framework" is currently being developed. Metadata will accommodate information needed for the preservation framework. Develop a modernized system with automatic workflow that can streamline the digital archive business process. XML is used for both metadata and business objects Manage workflow Create and maintain information (metadata) about digital objects Provide the ability to transform objects Provide representations of objects Build on a SOA Framework XForms for creation of Business Objects (Orbeon) Redesign and enhancement of archival metadata (ACE) XML Repository for Business Objects and Metadata (MarkLogic Server) Enhanced workflow management (SoftwareAG) Search capability NARA's ERA Systems Engineering developed proof-of-concept prototypes before new technologies were recommended for I3. New technologies were introduced into ERA for various reasons: JSP pages had thousands of lines of code and were fragile. Transformation required for getting data in/out of form. The current ACE design and implementation is unsatisfactory. Not self-describing Relationships are vague Contain minimal data Simple changes to business objects are difficult and costly. The current ECM doesn't support XML object well. Objects cannot be deleted No search capability Workflow currently hardcoded in ERA. Workflow changes require major recoding. Currently no search capability in ERA.

Organizations understand the importance of metadata. There are numerous metadata standards and implementations of metadata. NARA looked at many different standards (including their own) before defining the metadata (ACE) structure. This section will discuss the ACE concept and vision Metadata is data about data. There are many different metadata standards Dublin Core METS MODS EADS PREMIS I3 is a complete rework of the original ACE for IOC. The new ACE builds on the constructs of PREMIS. It includes NARA standard metadata that is currently being used for archival descriptions. LCDRG NARA's standard for Lifecycle Metadata LCDRG used for Archival Research Catalog (ARC) http://www.archives.gov/research/arc/ Contains metadata elements, as well as authority lists LCDRG available on-line at: http://www.archives.gov/research/arc/lifecycle-data-requirements.pdf

LCDRG Structure

All assets in ERA must contain an ACE entry. Business ObjectsThe documents that control the communications between NARA and contributing entities (government, collection holders, private donations) Government RecordsPermanent records that sent periodically to NARA for safe-keeping DonationsPrivate donations from organizations and individuals The Archival Research Catalog (ARC) is the online catalog of NARA's nationwide holdings in the Washington, DC area, Regional Archives and Presidential Libraries.http://www.archives.gov/research/arc/about-arc.html

Information in ARC

Archival Descriptions (currently 4 million entries) ARC descriptions cover mainly non-electronic records Intellectual metadata content written by archivists Representations of paper documents are available via NARA and NARA partners, i.e., footnote.com. Authority Lists Thesauri Data Geographic Subjects People Topical Subjects Organizations

NARA Archival Hierarchy

ERA is the National Archives and Records Administration's strategic initiative to preserve and provide long-term access to uniquely valuable electronic records of the U.S. Government, and to transition government-wide management of the lifecycle of all records into the realm of e-government.NARA Start with OAIS reference model Based on PREMIS model Incorporates LCDRG constructs that can be automatically populated on ingest.

Metadata for Increment 3

In I3 all current ARC descriptions will get an ERA ACE metadata description.

Challenges

Conversion of LDM to XML Schema NARA considers LDM to be normative version of the data model Keeping LDM and XML schema is sync after initial migration

LDM Snippet

Complete LDM Model

The reason why the universe is eternal is that it does not live for itself; it gives life to others as it transforms Lao Tzu NARA ERA Systems Engineering is continuously analyzing where the ERA system is at the current time and where it needs to be when it is fully operational. They analyze data and software technologies to improve efficiencies within the ERA system. Analyze the feasibility of XML Repository for ACEs and Business Objects Analyze the availability of software for native XML storage and retrieval purposes Standards-based approach Proof of concept was initially performed within NARA Systems Engineering * The proof-of-concept will be demonstrated. Based on NARA's proof-of-concept ERA systems integrator performed their own proof-of-concept. Data source - ARC XML files 4 million ARC records ARC Desciptions Thesauri Data MarkLogic Server Use Application Developer for initial application (GOD BLESS the MarkLogic Application Builder Team) Customized the default application using XQuery OxygenXML used to edit Application Developer files via WEBDAV (Love Oxygen - GOD BLESS Oxygen Developers, as well) Fully-functioning ERA Metadata repository. ARC data was available and was effective in proving the availability of functionality. Not an interactive content management system that provides editing and updating of records. XML repository for metadata is appropriate. Initial analysis proved it was scalable for large scale applications. Relatively easy to develop and maintain. Robust built-in searching capabilities. Easily integrates other information repositories that are available on-line. Proof-of-concept only skimmed the surface of MarkLogic capability. On-line demonstration.

The following screen shots are provided in order to demonstration some of the proof-of-concept functionality. This is being provided in-case the internet is unavailable during the presentation. Facited Navigation Number of objects

Home Page

Link to Outside Image Repository