tools for document interoperability
This is the web diary for nfoWorks and realization of the Harmony Principles. Pursuing Harmony tracks nfoWorks research, analysis, specification, and implementation of tools for document interoperability. There is commentary on related activities that address conformance, interoperability, and harmonization of document formats.
The nfoCentrale Blog Conclave
nfoCentrale Associated Sites
Also from the Cover Pages: XML Daily Newslink for 2008-10-14, there is word of jCAM, an open-source, Java-based CAM XML Processor and Template Editor. The SourceForge Project has the software; the web site has more information and tutorials.
Some of the lingo is opaque to me, but I do have interest in automated approaches that involve
I may have problems in the validation and filtering of documents and test suites that are not at the proper scale for CAM. I do think it is worth examining for ideas and applicable techniques. This is my placeholder reminder for that.
Technorati Tags: interoperability, IE8, web site construction, web standards, compatibility, conformance, document preservation, document formats
Be prepared for a dramatic shift in the reality of web-site browsing and the honoring of web-page standards. The pending release of Microsoft Internet Explorer 8 is going to put the reality of web standards and their loose adherence in our faces. Although Internet Explorer is indicted as the archetypical contributor to disharmony on the web, Internet Explorer 8 is going to challenge all of us to deal with the reality of our mutual contribution to the current state of affairs.
Here is a lesson, probably many lessons, for document interoperability and the way that standards for document formats evolve and harmonize, or not, over time.
The Web as Clinical Science
The movement from loosely-standard pages and their browsing to strictly-standard pages and standards-mode browsing will illustrate every aspect of the same challenge for office-productivity documents and the office suites that process them.
Web pages are the experimental drosophilae of digital documents. All aspects of dynamic convergence on standards, themselves evolving, and the forces of divergence, are demonstrated clearly and rapidly. I expect it to take Internet generations for significant convergence, with no static level of standards adherence anywhere in sight. It took us almost 20 years to get to this point on the Web; I figure it will take at least five more to dig out of it far enough to claim that there is a standards-based web in existence and in practice. I'm optimistic, considering that HTML 5, the great stabilization, is not expected to achieve W3C Recommendation status until 2012.
No document-interoperability convergence effort is anywhere close to the promising situation of the web as Internet Explorer 8, HTML5 implementations, and other compatibility-savvy browsers roll out over the next several years. It is useful to use that situation to calibrate how convergence and interoperability could work for document interoperability. There are significant technical barriers. The non-technical barriers are the most daunting. That should be no surprise.
Versioning in Document Use
I've written on Orcmid's Lair about the IE 8.0 Disruption. This involves changes in Internet Explorer 8.0 by which web pages are rendered in standards-mode on the assumption that pages are conformant with applicable web standards. In the past, it was presumed that pages were loosely-standard and browsers, also loosely-standard, made a kind of best effort to present the page. The consequences have been explained marvelously in Joel Spolski's post on Martian Headsets.
We are similarly relying on document-format standards as a way to provide for many-to-many interchange and interoperability between different (implementations of versions of) document-format standards and different (implementations of versions of) processors of those digital documents. That means we have a version of the loosely-standard documents with loosely-standard processing problem. We can't be strictly standard because the standards can't (and definitely don't) have strict implementations at the moment; and there are many ways that specifications and implementations have been kept loose by design. Accompanying that looseness by design is the the simple fact of immaturity among the contending document-format standards for office applications, particularly as vehicles for interoperable applications.
For office-productivity documents as we know and love them, there are five, count 'em five "official standards."
The "Official" Public Standards of Office Documents
For Office Open XML Format (OOXML), there is the ECMA-376 specification of December 2006. There is also the ISO/IEC 29500:2008 Office Open XML File Formats standard once it is made available. IS 29500 will have some substantive differences from ECMA-376. We won't have a solid calibration of the differences until the IS 29500 specifications are available and subject to extensive review.
For the OpenDocument Format, there is the Open Document Format for Office Applications (OpenDocument) v1.0 OASIS Standard issued 1 May 2005. There is also the ISO/IEC 26300:2006 Open Document For Office Applications (OpenDocument) v1.0 standard (also on the publicly-available listing). IS 26300 is for the same format as the OASIS v1.0 standard, but it is on a completely-separate standards progression. Appendix E.3 accounts for the differences of IS 26300 from the text of the May 2005 OASIS Standard. The first page of the IS 26300:2006 document (page 5 of the PDF) identifies its source as Open Document Format for Office Applications (OpenDocument) v1.0 (Second Edition) Committee Specification 1, dated 19 July 2006, derived from document file OpenDocument-v1.0ed2-cs1.odt; this is not another OASIS Standard, however.
The second and latest OASIS Standard for ODF is Open Document Format for Office Applications (OpenDocument) v1.1 issued 2 February 2007. This document is derived from OpenDocument v1.0 (Second Edition) Committee Specification 1, the same specification that is the source of content for ISO/IEC 26300:2006. The changes made to arrive at ODF v1.1 from the v1.0 (Second Edition) committee specification are detailed in Appendix G.4. There are some mildly-breaking changes from ODF v1.0 to ODF v1.1, mostly of a clarification or correction nature. There are a few additional features that have no down-level counterparts in ODF v1.0.
A third OASIS Standard, ODF v1.2, is under development. The current drafts, using a very-different organization from v1.1, are available as pubic documents of the OASIS Open Document TC.
We can expect to see more versions of ODF and of OOXML at their various standards venues. We'll be watching here on nfoWorks as the situation becomes even more chaotic. Notice that this diversity ignores the variety of divergent implementations of the various specifications.
Format Versions that Live Forever
It is possible for one document-format specification to officially supplant another, with the older specification deprecated. That has not been done so far with any of the five-and-growing document-format specifications, any more than it has been done for most of the versions of HTML specifications that have been recommendations of the W3C (and IETF before the development track moved entirely to W3C).
For example, the last full-up specification for HTML, the HTML 4.01 W3C Recommendation of 24 December 1999, has this to say about its immediate predecessor: "This document obsoletes previous versions of HTML 4.0, although W3C will continue to make those specifications and their DTDs available at the W3C Web site." This was possible because HTML 4.0 was young and there were important defects that 4.01 cured.
The HTML 4.01 specification continues with the following recommendation: "W3C recommends that user agents and authors (and in particular, authoring tools) produce HTML 4.01 documents rather than HTML 4.0 documents. W3C recommends that authors produce HTML 4 documents instead of HTML 3.2 documents. For reasons of backward compatibility, W3C also recommends that tools interpreting HTML 4 continue to support HTML 3.2 [W3C Recommendation 14 January 1997] and HTML 2.0 [IETF rfc1866 November 1995 and the IETF-obsoleting rfc2854 June 2000] as well."
The XHTML branch of specifications, originally derived from HTML 4.01, were intended as the basis for a future generation.
Meanwhile, there has been work toward both XHTML 2 and HTML 5.0.
HTML 5.0 is currently intended to exist alongside XHTML 1.x and its newer arrangements while also absorbing XHTML 1.x to some degree (by having an XML form). The current HTML 5.0 draft specifies legacy processing (in its HTML-syntax form) for variations of over 60 HTML DOCTYPE DTD flavors, extending back to HTML 1.0 and other variants. The intention is to converge HTML and XHTML 1.x under a consistent HTML 5 processing model with only no-quirks, some-quirks, and quirks modes. This is also intended to end the variation and extension of HTML (not XHTML) by capturing <!DOCTYPE HTML> for its own and having a concrete HTML syntax that is fully-divorced from both SGML and XML. It is important to point out that HTML 5 is not going to eliminate the divergence that browser (user-agent) plug-in models, plug-in implementations and scripting systems (especially client side) bring to the mix.
Document-format versions are not easily abandoned. Even if production of a format is deprecated, consumption of the format may need to continue into the indefinite future, and certainly so long as emitters of deprecated formats have significant usage. The W3C progression of HTML is at a point where that is fully-recognized and being honored in reaching toward an HTML 5 plateau sometime in the next decade.
Considering this promising stabilization, when would I manage to change all of my web sites and blogs to clean HTML 5 pages? Not until I know that visits to those sites are only a small fraction of Internet Explorer versions prior to IE8 (or maybe IE9) and other browsers lacking full-up standards-mode processing. Fortunately, the HTML 5 specification-effort promises to show me exactly how to do that in a mechanical way. I am looking forward to automated assistance. In my case, I'll also have the benefit of my IE 8.0 mitigation effort. Other web sites may require other approaches, and user browser choice will involve important trade-offs for some time.
I am surprised by the number of people who operate multiple browsers. Although I operate multiple products for office applications these days, that's mostly to explore their interoperable use, not to ensure ability to interchange documents (well, not until I joined OASIS and the ODF TC). I've been a serial adopter of Internet Explorer versions since IE 2.0. As a typical late-adopter, I may finally branch out now just to have a better calibration of the migration to standards-based sites and browsers for them.
This is an important lesson for the management of the expanding variety of specifications of formats for office-application documents, formats of which HTML packagings are sometimes one of the flavors.
Reconciling office-application document-format versions does not promise to be so easy as the current effort to stabilize HTML for the web.
The Looseness of Document Specifications
Of course, OOXML and ODF are not close dialects off a single family tree, as HTML variants might be treated (and HTML 5 demonstrates, if successful). In addition, the current specifications are not for same-conformance, interchangeable-everywhere documents:
Prospects for Interoperable Convergence
We already have before us difficulties with interoperable convergence of individual progression of a single standard and its variety of implementation. This makes the prospect of harmonization between different standard formats rather murky.
Desktop office-application software has more promise with regard to application of Postel's Law, to be liberal in what is accepted and conservative in what is produced. Unfortunately, the current specifications do not require conservative, interoperable implementations; the current specifications are arguably antagonistic to such an achievement.
I suspect that this is an unintended consequence mixed with some inattention to what it takes for interoperability to be achievable.
It remains to see how our experience and understanding matures. We are at the beginning, not the finish. The journey may seem endless.
The process of IE 8.0 mitigation and preparation for a standards-mode approach to web browsing impacts this site and blog as well as every other web page I have ever posted (somewhere over 120MB worth and climbing).
I'm not going to say anything more about IE 8.0 mitigation and HTML harmonization here. The overall effort will be tracked in that category of Professor von Clueless posts; that's the place to follow along. The lesson for document interoperability is something that is definitely appropriate for Pursuing Harmony; there'll be much more to say about that.
I can't stand some things about the sidebar, and I must fix them now (template versions 0.04-0.05):
I need to be careful and not attempt too much of this all at once. This is going to be one of those pages that I will probably update as I make progress.
Update 2008-08-20T23:54Z: A set of provisional changes are made with template version 0.04 with the idea of tweaking further after seeing the template at work.
Update 2008-08-21T01:07Z: The biggest change is to get rid of link underlining and use bold-face as well as an improved link color. This and some minor layout adjustments are accomplished with template version 0.05. I am going to let those sit there for a while until I see how to make the layout more pleasant still.
Update 2008-08-21T01:14Z: The changes work better if I save the new template to Blogger after previewing it.
Technorati Tags: interoperability, document formats, document processing, Harmony Principles, presentation conformance, conformance, ISO/IEC 24754
One problem for harmonization of document-format implementations in the case of ODF and OOXML is the degree to which those specifications may provide inadequate specification of how documents are to be presented in order for implementations to be adequately interoperable. This situation arises between implementations of the same format as well as between different document formats.
Along with establishing clarity on how much agreement in presentation is required, there is the small matter of being able to somehow confirm that an application achieves whatever that level of conformance and interoperability is.
As I suggested in "Interoperable ODF: Finding Ground Truth," there are many difficulties to be conquered in advancing from the current state of affairs to where there is reliable determination that implementations are substitutable in a particular interoperability setting.
We don't have good ways to even talk about the multiple, interlocking problems that lurk beneath the simple desire to have interchange of documents in standard formats "just work."
It would be helpful, for starters, to at least have a way to describe what a particular document-processing system does in rendering documents that it accepts. A checklist on the handling of particular features of an electronic document is also useful in determining conformance and interoperability guidance and, perhaps, eventual mechanical verification criteria.
The XML Daily Newslink for 2008-08-18 reports on a contribution that may be useful in this regard, the "First Edition of ISO/IEC 24754: Minimum Requirements for Specifying Document Rendering Systems." From the scope:
The Final Committee Draft (omitting the example in the informative appendix) is available as a 7-page, 74 kB PDF file.
Beside the announcement, here's what attracted my attention:
After having the template tweaks to look at, I am making some more, leading to version 0.03:
I've started capturing pages that illustrate the state of the blog at each template revision. These will be carried under the Blog Operations materials for reference.
Although I intentionally refrain from republishing the whole site, older archive and post pages will be regenerated from time to time. Page regeneration, always using the then-latest template, is usually because comments have been posted and sometimes because I provide additional information in an older post (such as links to related recent material).
While I am preparing for major additions and starting my homework as a new OASIS Open individual member, I am also performing some simple maintenance to the site template (resulting in version 0.02):
The new template previews correctly. Making this post will apply it to the main page, this individual post, and the archive page that this post is part of. Older posts have not been republished. They will continue to be formatted under the previous template, unless they are republished by me or as the result of a comment. I also backed up the blog to my development system before making the change on Blogger.
I believe there are extraneous cascading style-sheet (CSS) definitions in the <head> section of the template, and I will distill it down gradually. It is nice that there are no external style sheets, but I want to simplify what is sent to the browser even more.
The development of the fire drill and incident-handling materials is going to go slowly, and I will make more template changes when I need a respite from that and from content-creation activities.
The next series of activities for creation of this blog involve creation of a safety net against my own slip-ups and as a defense in case the blog becomes corrupted at some point.
This work takes up where Ramping Up has now left off. Compulsive knot-hole peepers will find a Construction Zone and Construction Notes on "Pursuing Harmony" Blog Operation.
1. First steps 2008-08-13T23:10Z. The first step is to capture the blog template and keep it in a form that is available under version control any time it is necessary to restore an earlier template:
2. Template Cleanup 2008-08-14T00:11Z. The next step is to clean up the template a slight amount in preparation for further customization:
3. Safety Net Incorporation 2008-08-13T01:23Z. With a maintainable template backed-up and versioned, there are many improvements called for. Those are cosmetic and not urgent, however desirable. Instead, provisions for locking down the blog, repairing/recovering it, and restoring it to operation will be put in place. The arrangement depends on the site being accessible by the administrator even though the blog is not working properly for some reason.
These pages are developed as part of the "Pursuing Harmony" Blog Operations material. That's the next step. Then we can have our first fire drill.
[update 2008-08-14T14:24Z added an important omitted word. I'm now back on the out-of-sight work needed in order to perform a successful fire drill. I forgot how tedious this is. The previous setup on Orcmid's Lair is old and stale and needs to be refreshed as part of that site's eventual repaving. Meanwhile, onward ... .]
This is an initial post for ramping up the Blogger creation of Pursuing Harmony. The blog will replace the manually-created diary that was first used with nfoWorks. This initial page is created to cause population of the blog, its archives, and the RSS feed.
The blog starts out using Jason Sutter's "Sand Dollars" format, one of the supplied Blogger templates. I will customize that to achieve my preferred style once operation of the blog is confirmed.
Customization happens after blog editing and posting is confirmed to be functioning.
My first problem was attempting to use SFTP. I reverted back to plain-old FTP to make sure the FTP log-in could succeed. This wasn't working the same as previously, so I tried a different path name. The problem may have been a lag in set-up of the account that I created for Blogger to use in posting to the web site. After one false-start, it all started working.
1. Update 2008-08-12T21:55Z After succeeding with this initial post, I decided not to use the Blogger NavBar. If I want to add a search panel for searching the blog and/or the site, I will do that later.
On getting Windows Live Writer working with the site, I attempted to establish an upload directory for images included in the blog. LiveWriter reported that it could not confirm the URL. I will make sure the new directory is established and give it a construction structure. I should then be able to have LiveWriter use it.
I am also going to move the archive directory out of the top-level blog directory, so that backup and maintenance is easier.
Live Writer reminded me to add tags to this post, and I will do that now too. I see that the categories work differently and are independent of tags. I must not be so careless in choosing categories.
2. Update 2008-08-12T23:34Z The archive is moved and the image directory was pre-populated with an index.htm so that Live Writer's concern for the folder was removed. It is now time to demonstrate that picture uploading works:
3. Update 2008-08-13T00:03Z After I figured out what time it is here (for adjusting to GMT in these little update items), I also managed to correct the URL for where the images are so that, not only are they uploaded, they are also correctly-reachable from the blog post.
My next effort is to make sure that I have a back-up procedure for mirroring the blog content on my development machine along with the other parts of nfoWorks.
After that has concluded, I will need to start working on the template.
4. Update 2008-08-13T00:12Z Well, the Neighborhood Rose images are on the blog site, but Live Writer fails to notice that I corrected the URL in my FTP settings. So I am linking in the above picture manually. I will introduce an additional image so that it can upload with the new settings.
Now I can introduce the backup procedure and then start adjusting the template.
5. Update 2008-08-13T00:53Z One concern I had is the automatic creation of subfolders to hold images from separate posts. I have verified that no subfolder index is available on the server and there is no way to explore the images but via image elements on the blog pages. That is satisfactory.
6. Update 2008-08-13T20:25Z I have carried out one manual backup. Before taking the customization to the next level, I have made a couple of simple touch-ups.
The next step is to construct the safety net and fire-drill procedure for the blog. It has been some time since a blog has been corrupted, but that is no reason to start operating without a net.
created 2008-08-13-18:06 -0700 (pdt)