Computer Science Department,
Rutgers University,
New Brunswick, NJ 08903
Bell Communications Research,
445 South St., Morristown, NJ 07960
shklar@cs.rutgers.edu
A side-effect of the universal acceptance of the World-Wide Web
is an urgent need to provide Web access to the vast legacy
of existing heterogeneous information. This information ranges from
documents in a variety of proprietary (and sometimes obscure)
representation formats to engineering and financial databases, and
often may only be accessed through specialized vendor tools and locally
developed applications. Moreover, rapidly increasing sophistication in
presenting information on the Web is already forcing us to treat
ftp and gopher information sources, and even early
HTML pages as parts of the same legacy.
In this tutorial, our main objective is to describe the current state of technology rather than prospective work. We discuss existing tools, methods and architectures, but we also mention functioning prototypes. To date, the main efforts in providing Web access to legacy data have centered around the following directions:
HTML and
Java graphic
user interfaces for existing applications
[mil95, rfd95].
In Section 2, we discuss the enabling technologies
for building gateways to non-Web applications.
In Section 3, we discuss
methods and tools for providing direct Web access to existing heterogeneous
information.
We believe that the most promising approach is building logical data
models and using them to support all kinds of sophisticated
presentation of the original information on the World-Wide Web.
Section 4 describes current solutions to the problem
of building Web interfaces to stateful information systems. We discuss
a number of popular tricks that provide partial solutions. The emerging
mobile code systems should help us to do better than that in the near
future. In Section 5, we discuss some of
the common problems that arise when using
Data presentation on the Web is performed by the Web browsers and
controlled by Fig. 1.
The
In (fig. 2), we present a sample Perl
[wal92] function for generating an
In (fig. 3), we present a sample
Perl function serving to establish a TCP/IP connection
with a remote application. The parameters to the function are the
address of the host running the application and the port number
($app_addr and $app_port). After completing the
socket binding but prior to attempting a connection, the function
sets the alarm clock, which aborts the attempt if it takes longer
than 30 seconds. The selectHTML to build
graphic front-ends for legacy applications. We believe that the mobile
code systems should solve many of these problems.
A brief summary of the overview and our conclusions are presented in
Section 6, followed by acknowledgments.
2.0 Enabling Technologies
HTML. When
presenting legacy data, HTML
has to be generated dynamically by a gateway program that either
accesses the existing data directly or establishes a
TCP/IP (or any other) connection with
a legacy application.
Interaction between gateway programs and Web servers is supported by the
Common Gateway Interface
(CGI) mechanism
(fig. 1).
CGI and
NPH/CGI mechanisms.
CGI mechanism supports passing
information between the gateways and the browsers via
HTTP servers.
Output information, generated by a CGI
gateway, is passed on to an associated HTTP
server that adds an HTTP header
and passes everything to the browser (fig. 1).
Special No-Parse-Header
(NPH) gateways pass their outputs
directly to the browsers. Programmers are responsible for building
HTTP headers (fig. 1).
This provides the greatest flexibility and is not much of a chore.
With the early versions of CGI, the
NPH/CGI mechanism
was the only way for programmers to influence the treatment of information
by the Web browsers by assigning the MIME
[bor93] types, specifying error conditions, etc.
Recently, CGI programmers have been
given limited control over the generation of
HTTP headers by the Web servers.
Still, NPH/CGI gateways provide
slightly better performance and allow full control over the generation
of HTTP headers, which may be important
for passing meta-information, etc. For more information on the
CGI mechanism see
[far96].
sub PrintHeader
{
local($type, $code) = @_;
$type = "text/html"
if (!defined($type));
$code = "200 OK"
if (!defined($code));
$ENV{SCRIPT_NAME} =~ m/([^\/]*)$/;
print "$ENV{SERVER_PROTOCOL} $code\n";
print "Server: $1/$ENV{SERVER_SOFTWARE}\n";
print "Content-type: $type\n\n";
}
HTTP header from an
NPH/CGI gateway. The function takes
two parameters: the desired MIME
type and
status code. The status code may determine the
success of an HTTP request, specify
a client action needed to satisfy the request, or provide a cause of failure.
The first two Perl statements serve to assign default values and the third
is used to extract the name of the gateway (referenced in the second
print statement by $1). Note our use of the
SCRIPT_NAME,
SERVER_PROTOCOL and
SERVER_SOFTWARE environment variables.
require 'sys/socket.ph';
sub Connect
{
local($app_addr, $app_port) = @_;
local($thishost,$name,$aliases,$proto,$type,$len);
local($thisaddr,$thataddr,$this,$that);
chop($thishost = `hostname`);
($name,$aliases,$proto) = getprotobyname('tcp');
($name,$aliases,$type,$len,$thisaddr) = gethostbyname($thishost);
($name,$aliases,$type,$len,$thataddr) = gethostbyname($app_addr);
$this = pack('S n a4 x8', &AF_INET, 0, $thisaddr);
$that = pack('S n a4 x8', &AF_INET, $app_port, $thataddr);
if (!socket(FS, $AF_INET, &SOCK_STREAM, $proto))
{
&PrintStart("Gateway ERROR");
print "The socket call failed.\n";
exit $!;
}
if (!bind(FS, $this))
{
&PrintStart("Gateway ERROR");
print "The bind call failed.\n";
exit $!;
}
$timeout = 30; # connection timeout in seconds
alarm($timeout);
if (!connect(FS, $that))
{
&PrintStart("Gateway ERROR");
print "Connect failed\n";
exit $!;
}
alarm(0);
select(FS); $| = 1;
}
HTML tags.
Another way of passing information between the gateways and the browsers
is by using the Server-Side Includes
(SSIs)
that force Web servers to parse documents and execute embedded
commands. However, letting users execute commands at the server
is a security risk, not to mention the cost of having the server parse
each requested HTML document.
Such cost may become quite significant under heavy load. Consequently,
few of the Web servers have the SSI
mechanism activated, further limiting its usefulness.
In our opinion, SSIs
are less secure, less efficient, and do not
present a viable alternative to the
CGI mechanism.
The most efficient way of interfacing a legacy system is to make
a gateway its own Web server. This job is greatly simplified by
the public availability of the
W3C
reference library,
which provides a general-purpose code base for implementing Web clients
and servers. The greater efficiency is due to eliminating
communications between the Web servers and application gateways.
This approach is the best choice for building commercial products though
the simplicity and flexibility of the CGI
mechanism make it preferable for research and prototyping efforts.
Fig. 4. Mobile code gateways.
The emerging
There have been numerous attempts to provide partial remedies for data
heterogeneity by implementing a variety of ever-changing
filters for format conversions. The filters are used to generate
Using the Multipurpose Internet Mail Extensions
(
The
There have been a number of attempts to build logical models of
distributed heterogeneous information and use these models to support
advanced Web presentation.
The Multimedia-Oriented Repository Environment
(
Folio InfoBase
from Folio Corporation is a commercial
product that includes a specialized Web server and development tools.
Their approach is to provide a proprietary hypertext format and to support
importing plain text and images in a number of popular formats.
Folio's product provides filters for the presentation-time
conversion of stored hypertext documents to
Fig. 5. Simplified Harness Architecture.
The objective of the
Closely related to this effort is the work on defining an Information
Repository Definition Language (
As in [per95], we define a legacy system
as a system that strongly resists modification and evolution.
Database management systems represent an important subset of legacy
information systems.
In Section 2, we described different ways
of interfacing independent gateway programs with the Web.
In this section, we discuss using such programs to provide Web
access to legacy information systems. We begin with discussing ways of
dealing with the statelessness of
The most common problems that arise when building Web interfaces to
legacy information systems are caused by the statelessness of the
The most popular solution is to either pass state and encoded authorization
information in gateway
In (fig. 6), we present a sample Perl
function that analyzes
The example in (fig. 7) illustrates the processing
of form fields. As mentioned earlier, hidden fields are often used for
passing state information - dynamic
Even assuming there was an acceptable way of maintaining state information,
there remains a more difficult problem of supporting transactions
that span multiple requests. Once the request is serviced, the
connection between the Web server and the browser breaks.
When a follow-up request comes through, the information system's thread
associated with the previous request may not be around. Starting a new
thread and retracing earlier steps within the same transaction to bring
the thread into a right state is not always possible, and quite
expensive when it is.
The Fig. 4. CICS Internet Gateway.
IBM DB2 WWW Connection Version 1 is now generally available
for OS/2 and AIX operating systems via a
download
from the Web. New beta versions for Windows NT, Sun Solaris and HP-UX
are also available from the same site.
Of course, it only makes sense to install this gateway product on
a machine that is also running a Web server and has a local
According to [lin95], IBM's current work is on
providing a Web
Other
Informix provides a free
CGI Interface Kit, which is a library designed for C applications
with embedded SQL. Application programmers are expected to build their own
Sybase's
web.sql is based on the
O2 Technology provides a
Oracle has announced the upcoming release of their
Oracle WebServer 2.0
A growing number of tools vendors have come up with Web interfaces for
accessing multiple DBMS products. The majority are
CGI-based (e.g., Saphire/Web from
Bluestone)
but there are also some that are either completely
SSI-based or employ some
combination of the CGI and
SSI technologies (e.g., Cold Fusion
from Allaire). Finally, there
are standalone Web servers specifically designed to support database
queries using the
The
Sapphire/Web is an interactive development environment
that supports the automatic generation of C and C++ code implementing
Cold Fusion is a development environment
for Windows NT and Windows 95, also designed to provide Web access to
relational databases that support Microsoft's
Open Database Connectivity
(ODBC)
interface. Cold Fusion utilizes both
WebBase
is a set of tools based on a database gateway
designed as a standalone
There is a variety of non-commercial tools available on the Internet.
They may be useful but any of them implies a fair amount of tinkering.
An interesting attempt to automate the process of building
Other freeware systems include:
Fig. 8. The Evolution of Computer-Human Interaction.
Apart from the purely technical concerns, the layout and functionality of
the
Problems with applying traditional interface design technics to building
interactive applications on the Web are rooted in limitations of both
Fig. 9. Menus in HTML.
Consider a sample legacy application with the user interface that utilizes
a pop-down menu. Selecting an item from the menu results in an immediate
action. Pop-down menus are also available in
A successful attempt of re-engineering the graphic user interface to
provide the Web front-end for an existing application is reported in
[rfd95].
One interesting result of this effort is providing additional functionality
that was not available from the original system. This, of course,
is due to the fact that the old graphic user interface did not take
full advantage of the available functionality. To summarize, even within
the limitations imposed by the use of the current versions of
We have discussed existing technology for building
Web interfaces to legacy information systems and for providing direct
Web access to legacy information. While
The World-Wide Web technology is advancing so rapidly that today's Web
data structures, including images, applets, and static and dynamic
Java technology
[gos95] provides another viable alternative
by letting programmers establish TCP/IP
connections from a mobile program executing at the browser
(applet) directly to a legacy application
(fig. 4).
The usefulness of this approach is much curtailed by the security
limitations that only allow applets to establish
TCP/IP connections with hosts
where they originated.
Another complication is that such applets are
responsible for all communications with end-users, and considerable
effort may be required in setting up even a simple prototype.
In such an architecture, programmers are completely on their own,
because they are not taking any advantage of the World-Wide Web other
than presenting the applet's graphic user interface through the
browser.
3.0 Direct Access to Heterogeneous Information
HTML documents either using the
CGI mechanism or off-line.
The off-line approach requires substantial human and computing
resources for the initial conversion and maintenance of information.
Maintaining the repositories presents the additional dilemma of
either creating new and updating existing information in
HTML, or
continuously managing evolving data in multiple formats.
The run-time approach helps to postpone the conversion until the information
is requested and eliminates problems with the initial conversion and
maintenance of information. However, the run-time conversion may not be
appropriate for some document formats (framemaker, etc.) for the following
reasons:
HTML
(may often require human post-processing).
MIME) [bor93],
supported by most Web browsers, helps to avoid data conversion
through the use of third-party presentation tools. However, it may require
renaming the original files because MIME's
type recognition mechanism relies on file extensions. Even though the
mapping of file extensions into MIME types
is customizable, it is still fixed for every given server, unless the type
assignment is performed by some specialized gateway program.
Adding support for new MIME
types often requires end-users to obtain and install third-party tools.
Further, there is still a problem of logically linking together
individual documents and of accessing arbitrarily formatted data.
OMNIS system
[cla95] has been
designed to provide access to library information that includes both
catalogs and digitized texts.
The scanned-in documents may contain images, postscript or other formatted
information, and are stored in a database. At presentation time, the
OMNIS
gateway converts textual information to
HTML, while images are converted
to common MIME types before
being passed to the browser. This is only feasible because
OMNIS has full
control over the format and representation of information that is stored
in its database.
Harvest [bow94]
provides support for extracting summaries from distributed heterogeneous
information and for executing searches over these summaries.
Once the resources have been identified, the responsibility of accessing them
is handed over to the Web browsers.
Harvest
provides efficient and flexible methods of indexing widely distributed
information. MIME mappings
are used to provide access to the wide variety of information and the
problems that were described earlier still persist.
MORE) [eic94]
was designed as a set of CGI programs
that operate in conjunction with a stock
httpd server to provide
access to a relational database containing meta-information, which is
used at run-time to retrieve physical data. The meta-information is
entered into the database off-line by the human librarians.
WebMake [bae95]
introduces methods for building Web structures over existing software,
e.g. source and object code for software systems.
In WebMake, meta-level
structural documents are used to create abstractions by
logically combining software modules or other structural documents.
A set of tools has been developed to provide a distributed
software development environment by utilizing the
CGI
mechanism. A specialized Web client is required to obtain full
access to the WebMake functionality.
HyperG [and95]
uses an object-oriented
database layer to provide information modeling and model maintenance
facilities in addition to integrated attribute and content-based search.
HyperG supports logical grouping of
documents into collections that may span multiple
HyperG servers. Special
cluster collections are used to group together related multimedia
and multi-lingual information. HyperG
uses its own HyperG Text Format
(HTF) that is converted to
HTML by the
HyperG servers when they respond to
HTTP requests.
HTML. Additional tools are provided
for designing customized query and navigation capabilities.
Harness
system [shk94, ssk95]
is to provide rapid access to large amounts of
heterogeneous information in a distributed environment without any
relocation, restructuring, or reformatting of data.
Like MORE and
HyperG, the
Harness
system uses metadata for search and retrieval of heterogeneous information
(fig. 5).
It provides advanced search and browsing capabilities without
imposing constraints on information suppliers or creators.
Harness utilizes a stable abstract class
presentation hierarchy, which need not be modified to add terminal
classes that accommodate new types of information and new
indexing technologies. Harness
provides tools for the automatic generation of meta-data based on user
inputs and the analysis of existing information.
IRDL)
[shk95] - a high-level language for describing
information resources
and the desired logical structure of information repositories.
The language provides high flexibility in imposing abstractions on
heterogeneous information.
Presently, the IRDL
interpreter generates Harness
metadata entities. With the emergence of
active objects, it should become possible to perform the
direct generation of the Web data structures.
4.0 Interfacing Legacy Information Systems
HTTP
servers. Next, we discuss Web access capabilities provided by
DBMS vendors. Finally, we discuss
Web freeware and third-party products that provide access to legacy
information systems.
4.1 Maintaining State Information
HTTP protocol and servers.
HTTP's statelessness creates obvious
problems in communicating with stateful information systems and in
supporting proper authorization mechanisms.
URLs, or as
hidden fields. This, of course, is only possible with dynamic
HTML pages. One trouble with this
solution is that it does not work with cached pages
that do not contain timely state and authorization information.
Most latest browsers either routinely re-evaluate dynamic
URLs or can be forced to always perform
the re-evaluation. Still, caching remains a problem.
A browser-specific alternative is based on the so-called
HTTP cookies. This method uses the special
Set-Cookie tag and requires that the
browser understands this HTTP tag and
uses it to maintain a persistent state (supported by Netscape).
sub ReadQueryString
{
local($queryString) = $ENV{'QUERY_STRING'};
local(%clientData);
#
# Decode URL and parse into local variables
#
# Change +'s to spaces, restore all hex values, and HTML special chars
$queryString =~ tr/+/ /;
$queryString =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;
# Load the '|'-separated positional values
($clientData{'action'},
$clientData{'server'},
$clientData{'port'},
$clientData{'query'},
$clientData{'state'}) = split('\|',$queryString);
return (%clientData);
}
CGI query strings
(CGI query string is a sequence of
characters following the first '?' in a CGI
URL). In this example, we assume that
query strings are composed of five '|'-separated positional parameters
('action', 'server', 'port', 'query', and 'state'). Note, that the query
string cannot contain spaces and binaries hence the
pre-processing to restore the original values. The results are accumulated
in the associative array %clientData, which is returned at the end
of the processing.
sub ReadFormFields
{
local ($key, $val, $buf_tmp, @buf_parm);
local (%fieldArray);
# Read in form contents
$timeout = 60; # the request timeout is set to 60 seconds
alarm($timeout);
$buf_tmp = "";
read(STDIN,$buf_tmp,$ENV{'CONTENT_LENGTH'});
alarm(0);
@buf_parm = split(/&/,$buf_tmp);
# Split form contents into tag/value associative array
foreach $parm (@buf_parm) {
# Split into key and value.
($key, $val) = split(/=/,$parm);
# Change +'s to spaces and restore all hex values
$val =~ s/\+/ /g;
$val =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C",hex($1))/ge;
# Use \0 to separate multiple entries/ 1 field name
$fieldArray{$key} .= '\0' if (defined($fieldArray{$key}));
$fieldArray{$key} .= $val;
}
return(%fieldArray);
}
HTML pages contain hidden fields with
values corresponding to the next state. The form information is read from
the standard input stream, the required input length is available from the
CONTENT_LENGTH environment variable.
As in fig. 3, we use the alarm function
to set the timeout. Elements of the @buf_parm array are
assigned individual fields that are processed within the foreach
loop. As in fig. 6, every field is pre-processed
to take care of spaces and binaries. The results are accumulated in
the associative array %field_array.
4.2 Commercial DBMS on the Web
CGI mechanism is widely
used by DBMS vendors for building
interfaces to their products. IBM's
DB2 WWW Connection [lin95] is
a CGI gateway using the
Distributed Database Connection Service
(DDCS) to establish and maintain
connections (for the duration of a request) with
DBMS that reside on
DDCS-supported platforms.
DB2 or DDCS
installation. With all the performance and
security shortcomings of the CGI
mechanism, DB2 WWW Connection Version 1 provides enough flexibility
and is a reasonable first step in providing Web access to IBM databases.
CGI interface to
their on-line transaction processing system, known as
CICSTM
(fig. 4, [lin95]),
which was designed to interact with
IBM 3270TM terminals.
The gateway captures the
3270 stream and converts it to
HTML. However, the problems of compromising
between the session-based CICS and
stateless HTTP protocols remain.
In addition, numerous invocations of CGI
programs (one per request) may create real performance problems.
The initial versions of the
CICS Internet Gateway for AIX 2.1 and OS/2 version 2.0.1 or later
have been officially released and are available as IBM products.
DBMS vendors supply their
own gateways to the proprietary database systems (e.g.,
Oracle
,
Informix,
Sybase and
O2 Technology).
Most of these products store the sign-up and state information
at the server, in their databases. This approach helps solve the
HTML page caching problem
but each access now requires an additional database query, and without
a reliable way of enforcing the sign-off procedure, new
problems of discarding and refreshing the state and authorization
information arise.
CGI gateways, and the use of the
library functions is illustrated by implementation examples.
However, it is not a product and a fair amount of tinkering is
likely to be required to get things going.
SSI mechanism
and supports embedding SQL statements into
HTML pages.
The
beta version of web.sql is available
for free evaluation. Sybase's evaluation license agreement prevents
the publication of comments on web.sql's
robustness and performance.
CGI
gateway for communicating with their object database applications.
There is certain confusion in that the
O2 Web Server is not an
HTTP server, but an O2
client capable of establishing connections with the O2
CGI gateway and converting results of
the Object Query Language (OQL) queries to
HTML. Nevertheless, the product is
fairly robust and sports nice object visualization capabilities.
TM, quite a few
steps beyond their current
CGI-based product. The new product is
supposed to include the following components:
The product is not yet available and it is impossible to say whether it
will live up to its expectations, but it is worth looking into once it
reaches the market.
TM, which is their
own http server that doubles as an Oracle gateway.
TM.
HTML-based server maintenance tools.
4.3 Database Access Freeware and Third-Party Products
ODBC or other mechanisms
(e.g., WebBase from
ExperTelligence).
CGI gateways that provide Web access
to relational databases. Currently, only
Unix versions of Sapphire/Web are available. The supported databases are
Informix, Oracle and Sybase. Bluestone intends to support other software
platforms and database systems in their future releases.
CGI and SSI
technologies. Applications are built by combining standard
HTML with high-level database commands
stored in customizable templates. In effect, application developers end-up
using a non-standard mark-up language.
HTTP server
that also provides access to ODBC-compliant databases.
The WebBase server does not guarantee support for all up-to-date functionality
of a stock HTTP server, which is not
a problem when Web sites are configured to use both the stock and WebBase
servers (on different ports) and when the WebBase syntax is used for defining
query templates. As with Cold Fusion, application developers end-up using
a non-standard mark-up language but the WebBase architecture provides for
better performance and simplified maintenance.
CGI interfaces is
described in [pfi95]. Authors have designed
a declarative Interface Definition Language
(IDLE) to support the generation of
their CGI-based
translation servers. Such translation servers may be used
for building Web interfaces to stateful legacy applications
(Section 4). A prototype version of the
IDLE interpreter is available.
CGI gateway contains
a form processing program and a customizable backend program for
proccessing SQL queries. The working
examples include interfaces Sybase, Interbase, Ingres and
Illustra/Postres databases.
CGI mechanism but is
designed specifically for Sybase databases. It's most notable feature
is that it provides a high-level schema notation for defining customized
Web presentation of database contents.
5.0 Web Front-Ends to Existing Applications
HTML front-ends to existing applications
alone may cause considerable anxiety on the part of end-users. In this
section, we comment on some of the Web-specific user interface issues.
A detailed discussion of the subject may be found in
[mil95].
HTML and the
HTTP protocol.
The HTML limitations were very
severe with the version 1.0, but have
been continuously improving with the introduction of
forms,
frames,
etc. (fig. 8, [mil95]).
HTTP
limitations make submitting requests similar to running a batch job:
data validation and error checking are only available after submitting
a request [mil95].
It is possible to define separate data validation requests,
but such requests are not greatly favored by the end-users
who have to wait for the validation results to submit a real request.
With Java, much of the data validation and error checking can be
performed at the browser, which mostly eliminates the problem.
HTML, but no action would follow the
selection. To retain the functionality, the
menu must either be complemented with a submit button or replaced
with a set of hyperlinks that point to menu items
(fig. 9). Any such choice should
be based on the application context. It would not generally
suffice to simply map legacy graphical elements into the ones that
are supported by HTML.
Instead, user interfaces should be carefully analyzed and redesigned
to make the best out of the available technology.
HTML
and HTTP, it is possible to design
successful graphic user interfaces for the legacy applications. However,
the flexibility of the emerging mobile code systems make them more
appropriate for the task.
6.0 Conclusions
Java and other emerging mobile code
systems provide new and exciting opportunities in presenting information
on the Web, we see them as complementary to the new developments in
the traditional HTML presentation
and the HTTP protocol.
HTML pages,
are likely to become obsolete tomorrow.
These structures represent a tremendous investment of money and talent
and can not be recreated with every new step in the technological advance.
Consequently, methods that support cutting-edge presentation of existing
heterogeneous information have to progress as well. Similar methods
should be applicable to creating virtual Webs of the future, with
both navigation and presentation controlled by personalized
meta-information.
7.0 Acknowledgments
I would like to thank Chumki Basu, Robert Farrell, Stavros Macrakis and
Louis Perrochon for comments and suggestions.
References