ELAG 2001
Report from Workshop 8:

XML and related technologies

By Jan Erik Kofoed, BIBSYS Library Automation

Background material

About 12-15 persons joined this workshop during ELAG 2001.
Some printed background material was prepared and handed out to the participants:

•	Crossnet: Using XSLT For XML MARC Record Conversion 
This is a report prepared for the ONE 2 project and describes methods for representation of MARC in XML and use of XSLT to transform between heterogeneous MARC formats.

•	Kofoed, Jan Erik: XML and friends
–	Part 1: XML and DTD 
A description of basic features of XML and the definition and use of DTDs.
–	Part 2: XML Schema
 A brief introduction to XML Schemas based on “XML Schema Part 0 primer”

•	MARC records in OAI XML format
Suggestions for MARC representation in XML from the Open Archive Initiative.
OAI has defined a metadata harvesting protocol that uses XML and contains a XML Schema for generic MARC records.
 
•	MARC records in XML format from BIBSYS and DBC
Experimental MARC formats from BIBSYS Library Automation and Danish Bibliographic Centre. These formats are not generic, but type specific for BIBSYS MARC and DANMARC.

Topics discussed

The following topics were subject for our discussion. They will be explained in more detail below:
•	Why XML  ?
•	XML and electronic publishing
•	XML and cataloguing
•	XML and data base systems
•	Transformation of XML
•	Query languages for XML
•	DTD vs. XML Schema
•	XML and Z39.50
•	Programming skill needed?
•	Tools and utilities for XML

Why XML?

This is the fundamental question. Fortunately, it is easy to demonstrate that bibliographic data are well suited for representation in XML. MARC is itself an example of this 
•	Bibliographic data is structured;
XML is well prepared for describing structure.
•	Much bibliographic data has a tree structure, (for example MARC data).
XML is also tree structured.
•	Data structured in XML is well suited for exchange.
ISO-2709 has long been used for exchange of MARC records. It is a well-defined format, but is only readable for machines, not for humans. XML has the advantage that it is readable for both machines and humans.

Here is a simple example of a MARC record formatted in XML:

Example 1:

<?xml version="1.0" encoding="iso-8859-1" standalone="no"?>
<!DOCTYPE marc SYSTEM "http://nysa.bibsys.no/dtd/bibsys-marc.dtd">
<marc xmlns="http://www.bibsys.no/xmlns/" type="BIBMARC" utgave="1.0">
	<f001>
		<f001a>911534067</f001a>
	</f001>
	<f008>
		<f008c>ger</f008c>
	</f008>
	<f082>
		<f082a>261.7</f082a>
	</f082>
	<f083>
		<f083a>D 172</f083a>
	</f083>
	<f130>
		<f130a>Müssen Christen Sozialisten sein?</f130a>
		<f130b>Zwischen Glaube und Politik</f130b>
	</f130>
	<f245>
		<f245a>Müssen Christen Sozialisten sein?</f245a>
		<f245b>Zwischen Glaube und Politik</f245b>
		<f245c>hg. von Wolfgang Teichert</f245c>
	</f245>
	<f250>
		<f250a>2. Aufl.</f250a>
	</f250>
	<f260>
		<f260a>Hamburg</f260a>
		<f260b>Lutherisches Verlagshaus</f260b>
		<f260c>1977</f260c>
	</f260>
	<f300>
		<f300a>117 s.</f300a>
	</f300>
	<f700>
		<f700a>Teichert, Wolfgang</f700a>
	</f700>
</marc>

And here is a record using the OAI MARC schema:

Example 2:

<oai_marc
    status="n" type="a" level="m" encLvl="7" 
    catForm="a" xmlns="http://www.openarchives.org/OAI/1.1/oai_marc"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xsi:schemaLocation="http://www.openarchives.org/OAI/1.1/oai_marc 
                    http://www.openarchives.org/OAI/1.1/oai_marc.xsd">
<fixfield id="1">"   90178038 "</fixfield>
<fixfield id="3">"DLC"</fixfield>
<fixfield id="5">"19900814092959.1"</fixfield>
<fixfield id="8">"900724s1974    po af         000 0 fre  "</fixfield>
<varfield id="10" i1=" " i2=" ">
 <subfield label="a">90178038</subfield>
</varfield>
<varfield id="40" i1=" " i2=" ">
 <subfield label="a">DLC</subfield>
 <subfield label="c">DLC</subfield>
</varfield>
<varfield id="50" i1="0" i2="0">
 <subfield label="a">MLCL 90/00723 (Q)</subfield>
</varfield>
 <varfield id="100" i1="1" i2="0">
 <subfield label="a">Berthou, P. Y.</subfield>
 <subfield label="q">(Pierre Yves)</subfield>
</varfield>
 <varfield id="245" i1="1" i2="3">
 <subfield label="a">Le C&#x00E9;nomanien de l"Estr&#x00E9;madure 
                     portugaise /</subfield>
 <subfield label="c">par Pierre-Yves Berthou.</subfield>
</varfield>
<varfield id="260" i1="0" i2=" ">
 <subfield label="a">Lisboa :</subfield>
<subfield label="c">1973 [i.e. 1974]</subfield>
</varfield>
<varfield id="300" i1=" " i2=" ">
 <subfield label="a">168 p., 67 leaves of plates :</subfield>
 <subfield label="b">ill. ;</subfield>
 <subfield label="c">33 cm.</subfield>
</varfield>
<varfield id="653" i1="0" i2=" ">
 <subfield label="a">Stratigraphic geology;</subfield>
 <subfield label="a">Cenomanian deposits;</subfield>
 <subfield label="a">Portugal</subfield>
</varfield>
</oai_marc>

XML and electronic publishing

•	Expressions of works stored in XML can easily make basis for different manifestations such as:
–	printed copies
–	electronic versions in PDF or PostScript
–	versions for display on screens with different resolutions
The basic document is formatted in XML. This document contains no formatting information, only content. Formatting instructions are stored in separate files, mainly based on XSL. Then the 
XML document is processed in combination with XSL giving the formatted copy.

•	Metadata can be added directly to the document as part of the XML structure.
In many cases this can be simple metadata records in Dublin Core. The author himself can produce such metadata. A librarian can eventually add more data and complete the metadata.

•	The authors need training and encouragement.
To write in a structured manner demands more discipline from the author. But the result will be better. The motivation must be the benefit of making documents that easily can be converted into different physical manifestation. The same document can on the fly be converted into a printed copy or a version for displaying on screen.

XML and cataloguing

•	XML is well suited for representing MARC records.
The structure of   MARC records can easily be represented in XML. The XML records can be efficiently handled by computers to build indexes for searching and proper storage systems.

•	Two methods:
–	Named elements:
<marc><f100><f100a><f100e><f245><f245a>
Good for internal use in library systems.
See example 1.

–	Generic elements:
<fixfield id=".."> <varfield id=".." i1=".." ...> <subfield id="..">
Best method for exchange of data. It can replace use of ISO-2709.
See example 2.

XML and database systems
•	Several RDBMS (Relational Database Management Systems) has added support for XML (Oracle, IBM DB2 and others)
–	XML is stored in a single cell.
The whole XML document is stored in one field, normally of type BLOB (Binary Large Object).
–	XML is stored as a collection of tables.
The XML structure is split into a collection of tables. This can be done automatically or manually. This approach can be efficient for accessing parts of the XML structure, but demanding joins can be the result of complex queries.
–	Special DML for handling XML.
To be able to convert a XML structure into a database schema, some systems use a special data manipulation language that is suited for transforming XML into RDBMS tables.

•	Specialized "native" XML DBS.
One example is Tamino from Software AG.
–	XML stored directly in a special internal structure made for handling XML.
The database schema is also a XML document defined in a DTD. The inner structure of the database is optimised for handling queries in a best possible response time. Loading data can be time consuming dependent on the number and type of indexes. From a library point of view, this system is well suited as a IR (Information Retrieval) system. It might not be the best choice for a heavy transaction oriented system. 

Transformation of XML

•	There is need for transforming bibliographic records for exchange between systems.
Interlibrary loan is one example. NISO has published a “Draft Standard for Trial Use” for a “Circulation Interchange Protocol”, Z39.83.  It uses XML as a wrapper for the records. The Open Archive Initiative  (OAI) has defined a generic XML format for MARC records, again look at example 2.
•	Use of XSLT together with XML simplifies converting of data.
XSLT (Extensible Stylesheet Language Transformations) is made for transforming in an XML structure into another structure. This can be another XML structure or something different from XML. If we use XML for wrapping bibliographic data, XSLT can be used for transforming from one type to another. If we have NORMARC in XML, we can use XSLT to transform it into MARC-21.
•	XML + XSLT might replace programs for generating and interpreting ISO-2709.
ISO-2709 is much used for exchanging bibliographic data. ISO-2709 practically impossible to read for humans, and it demands specialized software for interpretation. It is much easier to express MARC records in XML without losing any precision. Specifications written in XSLT can then be applied to the XML records to produce the preferred manifestation, for example a ISBD card. This can be achieved using standard software packages. 

Query languages for XML

•	XQL – an early suggestion from W3C.
This was used in the first version of Tamino, but is not supported anymore.
•	XSQL – an extension of SQL for use with XML stored in Oracle DB.
Here is a simple example of a XQL query:

<?xml version = '1.0'?>
<xsql:query xmlns:xsql="urn:oracle-xsql" connection="demo">
      SELECT 'Hello World' AS "GREETING" FROM DUAL
</xsql:query>

The result returned is:

<?xml version = '1.0'?>
<ROWSET>
   <ROW id="1">
      <GREETING>Hello World</GREETING>
   </ROW>
</ROWSET>

It is mostly a method for wrapping (Oracle extended) SQL into a XML structure. XSQL has also support for using XSL.

•	XPath – a W3C recommendation for addressing parts of XML documents.
This example locates a patient, which is born before 1960 and lives in Bradford:

/patient[//born < '1960' and //city='Bradford']

XPath is used as the fundament of the query language in the current version of Tamino.

•	XML Query – work at W3C for developing a new query language for XML.
The XML Query:
”List books published by Addison-Wesley after 1991, including their year and title.”
might be expressed as:

<bib>
      {
         FOR $b IN document("http://www.bn.com")/bib/book
         WHERE $b/publisher = "Addison-Wesley" AND $b/@year > 1991
         RETURN
               <book year={ $b/@year }> 
                  { $b/title }
               </book>
      }
</bib>

And return the result:

<bib>
     <book year="1994">
          <title>TCP/IP Illustrated</title>
     </book>
     <book year="1992">
           <title>Advanced Programming in the Unix environment</title>
     </book>
</bib>

DTD vs. XML Schema

•	DTD is inherited from SGML.
A DTD (Document Type Definition) is the formal definition of a certain class of XML documents. DTD uses its own special syntax.
•	XML Schema  is a new W3C recommendation (of 2 May 2001).
They use XML syntax and have much more power to express rules and constraints for building XML documents.
•	DTD and Schema make it possible to validate XML documents.
A XML document is valid if it obeys the rules expressed in the DTD or Schema. Validating is done in a XML parser.

•	DTD and Schema both describe structure, but the syntax is quite different.
The difference can be illustrated with some examples:

Here is a small bibliographic record as a XML document. The part referring to the DTD is in bold.

<?xml version="1.0" encoding="ISO-8859-1" standalone="no"?>
<!DOCTYPE book SYSTEM "book.dtd">
<book id="133" language="nor">
  <author>Hamsun, Knut</author>
  <title>Markens grøde</title>
  <published>
    <place>Oslo</place>
    <publisher>Aschehoug</publisher>
    <year>1948</year>
  </published>
</book>

This document is referring to a DTD:

<?xml version="1.0" encoding="ISO-8859-1" ?>
<!ELEMENT book (author*, title, published*, note*)>
<!ATTLIST book id CDATA #REQUIRED>
<!ATTLIST book language CDATA #REQUIRED>
<!ELEMENT author (#PCDATA)>
<!ELEMENT title (#PCDATA)>
<!ELEMENT published (place+, publisher*, year?)>
<!ELEMENT place (#PCDATA)>
<!ELEMENT publisher (#PCDATA)>
<!ELEMENT year (#PCDATA)>
<!ELEMENT note (#PCDATA)>

If we change the XML document a little, and puts the schema reference in bold, we get:

<?xml version="1.0" encoding="ISO-8859-1" standalone="no"?>
<book id="133" language="nor" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:noNamespaceSchemaLocation="book.xsd">
  <author>Hamsun, Knut</author>
  <title>Markens grøde</title>
  <published>
    <place>Oslo</place>
    <publisher>Aschehoug</publisher>
    <year>1948</year>
  </published>
</book>

It is referring to a schema:

<?xml version="1.0" encoding="ISO-8859-1"?>
<xsd:schema xmlns:xsd=http://www.w3.org/2001/XMLSchema 
            elementFormDefault="qualified">
  <xsd:element name="author" type="xsd:string"/>
  <xsd:element name="book">
    <xsd:complexType>
      <xsd:sequence>
        <xsd:element ref="author" minOccurs="0" maxOccurs="unbounded"/>
        <xsd:element ref="title"/>
           <xsd:element name="published" type="publishedType" minOccurs="0"
                        maxOccurs="unbounded"/>
        <xsd:element ref="note" minOccurs="0" maxOccurs="unbounded"/>
      </xsd:sequence>
      <xsd:attribute name="id" type="xsd:string" use="required"/>
      <xsd:attribute name="language" type="xsd:string" use="required"/>
    </xsd:complexType>
  </xsd:element>
  <xsd:element name="note" type="xsd:string"/>
  <xsd:element name="place" type="xsd:string"/>
  <xsd:complexType name="publishedType">
    <xsd:sequence>
      <xsd:element ref="place" maxOccurs="unbounded"/>
      <xsd:element ref="publisher" minOccurs="0" maxOccurs="unbounded"/>
      <xsd:element ref="year" minOccurs="0"/>
    </xsd:sequence>
  </xsd:complexType>
  <xsd:element name="publisher" type="xsd:string"/>
  <xsd:element name="title" type="xsd:string"/>
  <xsd:element name="year" type="xsd:string"/>
</xsd:schema>

•	Although the schema example might look more complex, schema is much better for describing groups and alternatives, and especially data types, constraints and patterns.
•	The conclusion is: Time has come for changing from DTD to Schema.
Since XML Schema now has become a W3C recommendation, it is quite safe to move from writing DTDs to writing schemas. A growing number of XML parsers are now supporting XML Schema.

XML and Z39.50 

•	New version of Z39.50  is being developed that uses both XML and SOAP .
It is called Z39.50 Next Generation (ZNG). Formerly it was addressed as ZML (Z39.50 over XML), but ZNG is now the right name. The proposal is described in: The ZNG Initiative  
•	Z39.50 uses ASN.1 and BER as data format. This will be replaced by using XML syntax. SOAP (Simple Object Access Protocol) will be used to wrap records.

Programming skill needed?

•	Basic XML with DTD or Schema do not require any programming skill.
(An advantage to think structured and orderly, i.e. be an librarian!) XML is easy to learn, and there are several good tools available that simplifies the process of writing, verifying and transforming XML records. XML syntax is in large parts similar to HTML syntax. Experience from writing HTML will make learning XML faster.
•	XSL (XSLT) needs some skill in programming. XSL and some other members of the family (XPath, XLink, XPointer etc.) is more complex and demands experience from system and program development. The basic use of XSL can best be illustrated with an example:

Given this XML document:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="books.xsl"?>
<book-order>
  <book>
    <author>Ibsen, Henrik</author>
    <title>A doll house</title>
    <price>$16.00</price>
  </book>
  <book>
    <author>Kafka, Franz</author>
    <title>The trial</title>
    <price>$34.99</price>
  </book>
  <book>
    <author>Shakespeare, Wilhelm</author>
    <title>The tempest</title>
    <price>$23.50</price>
  </book>
</book-order>

And this XSL program:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:output method="html"/>
  <xsl:template match="/">
    <html>
      <head><title>Book Order</title></head>

      <body bgcolor="#d0d0ff">
        <!-- set up the table -->
        <table border="border">

          <caption>
            <h2>Books on Order</h2>
          </caption>

          <thead>
            <tr>
              <th align="left">Title</th>
              <th align="left">Author</th>
              <th align="right">Price</th>
            </tr>
          </thead>
 
          <tbody>

            <xsl:apply-templates/>

          </tbody>
        </table>
      </body>
    </html>
  </xsl:template>

  <xsl:template match="book">
            <tr>
              <td align="left" width="300">
                <xsl:value-of select="title"/>
              </td>
              <td align="left" width="150">
                <xsl:value-of select="author"/>
              </td>
              <td align="right" width="60">
                <xsl:value-of select="price"/>
              </td>
            </tr>
  </xsl:template>

</xsl:stylesheet>

Combining the XML and XSL file yields a HTML file, which displayed in a browser looks like:

Tools and utilities for XML

The list of good XML tools is steadily growing. Here are just a few of them:

•	DB2XML - free
–	http://www.informatik.fh-wiesbaden.de/~turau/DB2XML
•	SXQL - free
–	http://www.hatop.de/sxql.html
•	Microsoft XML Notepad - free
–	http://msdn.microsoft.com/xml/notepad/intro.asp
•	Apache XML Project - free
–	http://xml.apache.org/
•	SoftQuad XMETAL - commercial
–	http://www.softquad.com/xmetal/
•	XML Spy - commercial
–	http://www.xmlspy.com/