Programming

unAPI format and Semantic Web

2009-05-24T04:59:00.000-07:00

Ross Singer's recent post about "One Data Format Identifier (and Registry) to Rule Them All" caused some interesting discussions. I didn't read all of them, and don't have much to add, but I do have a chance of reflecting how unAPI format wiki was created at the first place.

It turned out that I always have trouble of understanding why Semantic Web is so obsessed with URI and Ontology. I am not Semantic Web expert, but to me the strict URI approach seems to directly conflict with other "convention over configuration" approaches like tagging, wiki, twitter. And it doesn't seem everyone will have the time to learn each other's ontology anyhow. As a result, any sufficiently large RDF file always make my head spin because of all these long URIs, and I am not sure how much bandwidth were used to carry them on the Internet.

So why don't just use a word as the identifier, and let everyone pickup a dictionary and find out its semantic. I guess Oxford dictionary is a better agreed-upon ontology. In retrospect, perhaps this is how unAPI format differs from SRU/OpenURL, we can choose a name and hopefully it will work. Does it work? I don't know.

Since multiple copies keep stuff safe, I also paste here the unAPI format retrieved from Internet Archive.

name type example desc doc
amazon application/xml opa a convention used by OPA
asn1 text/plain opa Abstract Syntax Notation One asn.1
bibtex text/plain hubmed bibtex bibtex
dc text/plain opa unqualified Dublin Core
didl application/xml TODO MPEG-21 DIDL didl
endnote text/plain refbase endnote endnote
latex application/x-latex refbase latex latex
marcxml application/xml Technosophia MAchine Readable Cataloging in XML marcxml
markdown text/plain refbase markdown markdown
mods application/xml Technosophia Metadata Object Description Schema mods
html text/html refbase HTML HTML
oai_citeseer application/xml opa
oai_dc application/xml Technosophia unqualified OAI Dublin Core oai_dc
pdf application/pdf refbase Portable Document Format PDF
pubmed application/xml opa pubmed article pubmed
rdf/xml application/rdf+xml hubmed Resource Description Framework RDF
ris text/plain hubmed ris ris
rss application/xml Technosophia Really Simple Syndication RSS
rtf application/rtf refbase Rich Text Format RTF
srw_dc application/xml Technosophia unqualified SRW Dublin Core
srw_mods application/xml refbase unqualified SRW MODS
text text/plain opa
wrap application/x-javascript opa unalog json format

cannonical URLs in tag

2009-02-27T07:04:00.000-08:00

Having observed a few canonical URL solutions in Digital Library world, such as Handle, PURL, DOI, it is really nice to see the cannonical URLs from Google. Google partially solves the problem by simply adding a <link rel="canonical" href=""> to specify your preferred version.

It always surprises me how elegant and simple a solution can be, and how long it takes to find these simple solutions.

canonical URL

2007-05-22T20:14:00.000-07:00

Just came across these URLs

http://www.google.com/search?hl=en&q=link%3Ahttp%3A%2F%2Fworldcatlibraries.org%2Foclc%2F65187392&btnG=Search

and

http://www.google.com/search?hl=en&q=link%3Ahttp%3A%2F%2Fworldcat.org%2Foclc%2F65187392&btnG=Search

I was trying to compare same Worldcat pages with different URLs, by google backward link search features (http://www.google.com/advanced_search)

So http://worldcat.org/oclc/65187392 and
http://worldcatlibraries.org/oclc/65187392
are same page, however due to variants of these URLs, Google treats them as different URLs, and this certainly doesn't help moving up these pages up in Google search results, if we consider the inbound link factor in PageRank.

new xISBN API - beta

2007-03-02T18:58:00.000-08:00

I have been working on a new xISBN service in last few months, and I had a chance of sharing latest progress in excellent code4lib conference held in Athens, Ga. Thanks for all your valuable comments, folks!

The new service is now running as a beta version. We add more features/metadata, at the same time try to keep the API simple and clean.

x-identifier list is a venue for comments/suggestions.

things learned from SOLR api

2007-02-27T20:25:00.000-08:00

things learned from SOLR api

I am fortunte to attend Erik Hatcher's one day Lucene/Solr tutorial, what's really attracting me is elegance of SOLR's REST API, which is at the same time concise and consistent. To start with:

-- add/query/delete/commit are all simple REST API.

as an example, delete a record is a simple HTTP POST "<delete><id>SP2514N</id></delete>", do I need to say more?

-- allows different format (but equavalent infoset)
"select?q=ipod&wt=xml" -- returns an xml format
"select?q=ipod&wt=python"
"select?q=ipod&wt=ruby"
"select?q=ipod&&wt=json"

They all return same infoset.

-- the way of controlling returned fields by query

q=video&fl=name,id (return only name and id fields)
q=video&fl=name,id,score (return relevancy score as well)
q=video&fl=*,score (return all stored fields, as well as relevancy score)

So user has control of which fields to return.

-- no xml namespace at all

This is subject to argue, but I tend to think in RPC-oriented application XML namespace doesn't really matter, and in Document-oriented applications XML namespace are important.

-- the way of organizing schema and result.

Basically everything is a field, a field has a datatype, name and value. Such as following result:

<str name="id">IW-02</str>
<bool name="inStock">false</bool>
<str name="manu">Belkin</str>

This bascially allows any fields without XML namespace/schemas.

google's moon shot

2007-01-30T20:34:00.000-08:00

New Yorker magazine runs an article "google's moon shot" about google book project. It mentioned Google aims to scan at least as many as worldcat records (32M) in 10 years. And it also mentioned the hate/love relationship between publishers and Google, and seems optimistic that the copyright issue can be resolved this way or other. There are also interesting thoughts of networking books, similar to the way of how web pages are linked together by hyperlink.

Ebook has been here a while and we are still trying to figure out the right way, maybe Google, Open Content Alliance, and other's work can make a big difference, we will see how this evolves in next few years

microsoft live book search with fulltext download

2007-01-20T07:52:00.000-08:00

I just have a chance of looking at Microsoft live book search demo , it's still a small collection, so search results can be confusing. However an entire book can be downloaded from archive.org as a PDF file, such as this one:

http://www.archive.org/download/writingsofcharle01dickuoft/writingsofcharle01dickuoft.pdf

I guess this is part of deal of Open Contents Alliance. Google Print doesn't really allow download of a full book.

convert between 13 digits and 10 digits xisbn

2006-11-07T14:33:00.000-08:00

self note.
(a) from 10 digit to 13 digits

if 10 digits takes the format ABCDEFGHIJ, 13 digits is going to be: 978ABCDEFGHI(?), and last digit (checksum) can be calcualted by:

(10-((9*1)+(7*3)+(8*1)+(A*3)+(B*1)+(C*3)+(D*1)...+(I*3)) modulo 10) modulo 10

(b) from 13 digits to 10 digits
if the first three digits is not "978", it cannot be converted to 10 digits.

if the first three digits is "978", takes remaining 9 digits: ABCDEFGHI, and calcuate checksum for 10 digits ISBN:
(11-((A*10)+(B*9)+ .... +(I*2) modulo 11)) modulo 11

10 will be replaced by X

reference:
[1] http://wiki.tcl.tk/12638
[2] http://www.isbn.org/standards/home/isbn/transition.asp

googlemap api and craigslist

2006-10-22T21:34:00.000-07:00

It's the right season to find an apartment again ;-) this time I am mainly using craigslist to look for apartment. Fortunately, most rentals have a google map link, but after a while, it becomes really difficult to keep track an interest list.

For a while I want to try the goolgemap API and this is the right time to do so, it turns out the API is really handy. I need to register a googlemap API account, after I just need follow some javascript code examples in googlemap homepage, in less than 2 hours I can conveniently compare interesting apartments through the nice google map interface.

I still need manually populate craigslist information into my small application -- it would be great if craigslist have a built-in function to create a "my bookshelf" kind of service.

oaiarc new release

2006-10-15T20:12:00.000-07:00

oaiarc-1.0 is just released, this release includes following changes:

--internationalization support (with English, Spansih, French built-in)
--strict XHTML compliant
--UTF-8 support
--remove oai 1.x support
--various bug fixes

Juan Corrales from Universidad Rey Juan Carlos de Madrid contributes
most improvements in this release.

pathways paper

2006-10-13T14:22:00.000-07:00

The last two years pathways research result are published:

An Interoperable Fabric for Scholarly Value Chains - D-Lib, October 2006 (link will be active early next week)

Pathways: Augmenting interoperability across scholarly repositories - International Journal of Digital Libraries

dump a web page and its links by wget

2006-10-13T14:14:00.000-07:00

self-note:

Every few weeks I need to read wget manpage to find parameters to cache a single webpage and all its links, so perhaps better write it down here to save 5 minutes:

wget -r -H -l1 -k -P $targetdir --exclude-domains ${comma-seperated domain name} --user=xxx --password=xxx $url

Poincaré conjecture and Shing-Tung Yau

2006-09-06T20:51:00.000-07:00

it was sad to read recent New Yorker story of Poincaré conjecture and Shing-Tung Yau. Yau won Fields award and has greatly contributed to mathmatics study, his opinion (in general) is very well respected and cited in China.

I believe this article must have exaggerated some information, as evidenced by clarifaction from some mathematicians recently, still, it is really difficult to know when to stop, even as brilliant as Yau. So this is really a sad and disappointed story.

notes in Fedora User Conference 2006

2006-06-25T21:46:00.000-07:00

I took some notes in the Fedora user conference 2006. Disclaimer: I am not a Fedora user; and the conference doesn't have proceedings and presentation is not online -- so read with caution.

The conference is nice but somehow difficult to summarize because many presentations are project-oriented. So I will try to summarize them in several perspective: (a) core functionalities (b) interesting project (c) some observations and thinkings.

(a) core functionalities

RDF triple store: A very interesting part of Fedora is its usage of RDF and triple store, as I have always been interesting in the scalability of RDF triple store. Dean B. Krafft of Cornell gave some interesting numbers about NSDL 2.0 usage of Fedora. NSDL 2.0 handles some ~2M digital objects, with 70 RDF tuples for each object, overall Kowari triple store has 163 Million triples.

Content model: The ongoing development of Fedora Core includes "content model", such as structure/ontology for thesis, article, etc, and dynamic dissemination associating with "content model". The question is how to reach agreement on common ontology.

Dissemination/behaviour: This seems like a pretty active area, it's about given a digital object, how can you associate services with this object. DLF Aquifer project has an interesting concept of "AssetActions", which defines an XML schema of associating behaviors with object, initial experiment based on Fedora.

(b) projects

This is perhaps the most interesting part of the conference, there are several ambitious, national project using Fedora as core service, including NSDL, German's eSciDoc, Aussie's ARROW, and DART, and some works by Harris Group. These projects have a theme of building workflow system for scholarly communication in a large scale. Although information can be overwhelming in some cases, it's really a good headstart to take a further look at these initiatives.

(c) thoughts

RDF store or RDBMS?

an old question, but still relevant and interesting

What's essential functionalities of a repository?

Is it OKI, JSR170, Fedora core API ?

A nice graph of describing repository:

Repository->services->applications->interfaces
---------------------------------------
when moving left to right, the changes are more likely, repository is expected to be stable.

New scholarly communication system

I think we definitely need some models and researches in this area, such as this and this

Liveclipboard copy beyond microformats -- unapi 1.0 released

2006-06-25T20:06:00.000-07:00

Dan Chudnov recently released unAPI version 1. Here I particularly want to compare unAPI with microformats in liveclipboard implementation.

The MS liveclipboard demo is based on microformats, essentially, the demo uses some smart javascripts to copy XHTML page between different web pages. In principle, the technology can be used to copy any XML fragment, or even binary files if you consider base64 encoding.

Microformats is well suited for this purpose because its XHTML fragments are well-formatted XML. However, there are many applications/domains are not covered by microformats: (a) not all contents can be described by XHTML, such as tremendous XML standard/contents in the web, or many non-HTML content in the web. (b) even a content can be described by XHTML, it may never reach the radar of microformats.org, or never have enough incentives to make them standard.

So here I think unapi can fill a gap, because it allows copy of any content, far beyond the scope of microformats.org, from initial implementation of unapi we saw mods, dc, json, pubmed, rdf, text, and we can expect more diversified formats in the future. All these formats may eventually become payload of liveclipboard, and the application is up to your imagination.

On the other side, the approach of unAPI is more complex than microformats. In microformats you can simply markup an XHTML page and it's all done. In unAPI one have to markup an XHTML page and implement a simple API. And the little tradeoff can be worthwhile if we want to take full advantage of liveclipboard.

Stuff I like in JCDL 2006

2006-06-14T16:07:00.000-07:00

some interesting readings in JCDL:

"Also By The Same Author: AKTiveAuthor, A Citation Graph Approach To Name Disambiguation. " Duncan M. McRae-Spencer, Nigel R. Shadbolt

This paper describes how to use author's self-citation (for good or bad, anyhow another topic) to implement name disambiguation. For example, if a paper has an author "X. Liu", and again I cite a paper with author "X.Liu", it's quite likely that the two "X.Liu" are same person. By taking advantage of this social context, the paper got starling precision/recall in name disambiguation.

"Bibliometric Impact Measures Leveraging Topic Analysis", Gideon Mann, David Mimno and Andrew McCallum.

Andrew McCallum wrote the popular bow/rainbow text classification package. This is the first time I saw his paper in JCDL. The paper is based on a new cluster method TNG, which can label clusters by phase, instead of individual words, e.g. "text classification" can be used to label a cluster, instead of "text" and "classification". This is extremely powerful in labeling cluster. After that the paper proposes several impact measures of topic, particularly life cycle of how a subject/topic emerges, develops, and influences other topics. Pretty interesting reading and solid work.

"Building a Research Library for the History of the Web", William Y. Arms, Selcuk Aya, Pavel Dmitriev, Blazej J. Kot, Ruth Mitchell, Lucia Walle

I blogged this work before, the project tries to mirror and mine whole Internet Archive. Anything dealing with that level of scalability is worth checking out.

"Metadata aggregation and automated digital libraries: A Retrospective on the NSDL experience", Carl Lagoze, Tim Cornwell, Naomi Dushay, Dean Ecktrom, Dean Krafft

Some remarkable lessons of OAI-PMH in a very distributed system. It also wins best paper award. Loosely distributed system is always a hard question, especially you are targeting transaction-level quality, no matter distributed search or harvesting.

"EcoPod: A Mobile Tool for Community Based Biodiversity Collection Building" YuanYuan Yu, Jeannie A. Stamberger, Aswath Manoharan, Andreas Paepcke

A PDA-based application for biology species observations, it is not complex or abstract, and it focuses on a simple task and solve it well. Maybe that's how the research should be done in many DL projects: do one thing and do it well.

"An architecture for the aggregation and analysis of scholarly usage data." Johan Bollen and Herbert Van de Sompel

It describes using OAI-PMH to harvest usage data, which is embedded in OpenURL ContextObject. There are also interesting result of mining these usage data. I think the choice of OAI-PMH/OpenURL are very much appropriate here.

unapi scripts, and a wiki page

2006-05-21T21:08:00.000-07:00

I updated several thing regrading unapi:

-- fix a bug in unapi userscript

-- update a proxy demonstration to unapi revision 3.

-- I started a wiki page in http://unapi.stikipad.com/, although a personal opinion, I think it is a good idea to document commonly used format in unapi, largely motivated by a similar microformats list. Feel free to update the wiki page if you have comments/ideas.

2006-05-19T13:32:00.000-07:00

unapi greasymonkey scripts updates to revision 3, the URLs are same so you may follow previous instructions at: unapilink,
citeseer and pubmed, and amazon.

2006-05-18T15:17:00.000-07:00

DIDLTools package online

The DIDL Tools is just released http://african.lanl.gov/aDORe/projects/DIDLTools/, as posted in release page:

"The aDORe DIDLTools is a Java toolkit for the construction, validation, serialization and de-serialization of the MPEG-21 DID data model. DID, the MPEG-21 Digital Item Declaration, provides an abstract model for the representation of digital items, whereas DIDL, the MPEG-21 Digital Item Declaration Language specifies how to serialize the model in XML. The API provided by the aDORe DIDLTools allows for the construction of customized DIDL XML documents, as well as provides flexible and extensible serialization methods."

There was some good thinking of allowing any metadata in the model, have a look and ask questions in http://lists.lib.ugent.be/mailman/listinfo/didwriter-dev

2006-05-16T14:26:00.000-07:00

slides of AISTI Mini-Conference

The AISTI Mini-Conference has its slides online, I was in the conference and there were several pretty good presentations.

Douglas Fils made a presentation about Chronos project, titled Services Architecture and Semantic Practices Using the CHRONOS Cyber-Infrastructure Effort, the project combines some cool web2.0 technology to build Chronos environment.

Alex Szalay presented Science in an Exponential World, which overlaps library world with data curating. I pretty much like his remarks that data must be close to analysis tool and we need journal for data. His work is also assocated with microsoft 2020 science initiative.

2006-05-10T21:32:00.000-07:00

arxiv and trackback

I recently listened Paul Ginsparg's interview by CNI, it touches history and ongoing development of arXiv.

This is the first time that I heard arXiv has trackback feature, see http://arxiv.org/help/trackback, as always, arXiv does everything differently (no kidding, I am a regular arxiv user), and seems there was a hot debate about its policy a while ago in physics blogsphere.

I then quickly checked hubmed and WPopac, two blog-style library service, it's no suprise that both support trackback.

However, arXiv is unusual, because its interface/functions is stable for quite some years, it shows what's quality and good design. Now it has a trackback, the fact shows quite nicely how new techologies can be integrated with established library service.

2006-05-06T18:16:00.000-07:00

WebCite

Just read WebCite from Lorcan Dempsey's blog.

WebCite is an archiving system for webreferences (cited webpages and websites). It works in a way of tinyurl, by submitting a URL you get a new URL back, e.g. I submit "http://www.yahoo.com" on May 6th, 2006, webcite gives me back a new URL http://www.webcitation.org/5Fgu0Xf5x, and yahoo page on May 6th, 2006 is archived in http://www.webcitation.org/5Fgu0Xf5x.

So this is actually a combination of archive.org and tinyurl.com, and may look extremely simple. However, as I worked on persistent URL and web archiving issues previously, I can testify simple is not easy.

Persistent URLs issues have been studied by Steve Lawrence (citeseer, now google) in (Persistence of Web References in Scientific Research. IEEE Computer, 34(2):26--31, 2001), Thomas A. Phelps and Robert Wilensky in (Robust Hyperlinks and Locations ), Frank McCown and Michael L. Nelson's "The Availability and Persistence of Web References in D-Lib Magazine".

In particular Thomas Phelps has proposed that most web pages can be uniquely identified by 5 keywords, so you should referene the web page by the five keywords, and issue the 5 keywords to web search engine to find the web page (in case it's relocated).

Web archiving is best addressed archive.org for now, in a rather typical web crawling way, and still an active research area. And now we have a on-demand web archiving system such as webcitation.org.

I don't know how webcitation.org scales, or how it protects the data, and it certainly doesn't solve all web archiving issues (we may only realize a web page should be preserved after the right time is long gone), but still, it's really simple, neat, and targetted two problems (persistent URL and web archiving) so nicely.

2006-05-01T15:55:00.000-07:00

learn tricks in java jar file

I used java jar files for some years but didn't realize several interesting mechanisms inside of jar file. Now I have a chance of reading jar specification and found several very handy features:

-- specify class-path in MANIFEST.MF

-- create index files (INDEX.LIST) for all packages.

-- META-INF/services directory to associate an implementation with its definition (e.g. class to interface)

These all come handy for easy distribution of a package

2006-03-22T20:57:00.000-08:00

Programmable Web and protocol

Some common web API design patterns are discussed by Raymond and Leigh. Following my previous blog, I am thinking there are different requirements in designing service and protocol. When writing a service, one has control of all resources and has the luxury of making everything clean and right, including all kinds of error processing, so no ambiguity here. While working on a protocol, perhaps it's ok to get minimal correct because one don't have control of all resources.

So perhaps this may help explain the difference between programmable web and Atom publishing protocol, and why Atom dropped detailed status code between 0.2
and 0.4

2006-03-22T07:42:00.000-08:00

Producer and Consumer, the economy of unAPI

While I am riding the bus to Los Alamos, I ponder a problem which annoys me for a while in unAPI design and was debating still now: what's simple, and not too simple in unAPI?

I think I got something and eager to write down, again this is an economy issue. There are two parties in most web applications: producers and consumers; when producers are potentially quite outnumber consumer, you want to make producer as simple as possible, and leave most complexity to consumer, because consumer benefits more ($$$) from this cycle and therefore is willing to afford the complexity.

Maybe this is too abstract, but it may make sense if we think about several examples. (a) In the web, there are so many nasty HTML pages, but we are all happily living with it, because the consumer (browser, search engine, etc) takes extra efforts of making sense of these pages. (b) In the RSS/Atom world, there are several standards and many more nasty pages but news aggregators happily support them all.

So maybe I have the answer now. In unAPI we want to make the producer as simple as possible, we don't want to burden them for heavy error processing or schema validation -- so we can claim they are wrong here and there, no! it must be extremely easier and nasty errors are tolerated (such as HTML or RSS), content is king here; it's up to the consumer to make order from chaos.

2006-03-12T13:27:00.000-08:00

unAPI proxy: power of URI

There has been a lot of debates the nature of "identifier" in unAPI. To demonstrate the power of URI, I put together an unAPI proxy http://dp9.cs.odu.edu/unapi/js/uproxy.html, which modified Alf's copy/paste example to demonstrate that URI indeed helps. In this demo, the javascript extracts responses from both Opa and Hubmed: multiple views of same resource. It's possible because both servers support info:pmid URI.

2006-03-09T09:10:00.000-08:00

How well do search engines index the OA repositories?

Frank McCown, Xiaoming Liu, Michael L. Nelson, Mohammad
Zubair (2006) Search Engine Coverage of the OAI-PMH
Corpus, IEEE Internet Computing, March/April 2006.
http://library.lanl.gov/cgi-bin/getfile?LA-UR-05-9158.pdf

Abstract: The major search engines are competing to index as much
of the Web as possible. Having indexed much of the surface Web,
search engines are now using a variety of approaches to index the
deep Web. At the same time, institutional repositories and digital
libraries are adopting the Open Archives Initiative Protocol for
Metadata Harvesting (OAI-PMH) to expose their holdings, some of
which are indexed by search engines and some of which are not. To
determine how much of the current OAI-PMH corpus search engines
index, we harvested nearly 10M records from 776 OAI-PMH repositories.
From these records we extracted 3.3M unique resource identifiers
and then conducted searches on samples from this collection. Of this
OAI-PMH corpus, Yahoo indexed 65%, followed by Google (44%) and MSN
(7%). Twenty-one percent of the resources were not indexed by any
of the three search engines.

2006-03-08T21:30:00.000-08:00

unapi and live Clipboard

This is a copy of email to gcs-pcs list, following Alf's copy/paste example, and live clipboard, it live demo, and screencast.

I am not sure how much I got live Clipboard, but looking at their demos really helps a lot, see screencam, so perhaps helpful to others too.

The last three examples are more advanced than microcontent copy, they are about copying RSS feeds URL, and in the pasted site these RSS feeds are dynamically loaded. Put it another way, the data is "live" in pasted site.

Well, I think this is associated with our discussion of unAPI here. It seems like we can use Clipboard to copy URI+unAPI baseURL, and, the data is also "live" in pasted site in two senses: (a) the pasted site can decide which formats to use (b) the pasted site can decide to use other unAPI baseURL.

I guess the question is about how to plug URI+unAPI into live clipboard, like the hcalendar or RSS feed. Not sure if there is going to be an API or have to read the source code, lots to learn here. But my initial impression is that they are compensative technologies, similar to the relationsahip between RSS and Clipboard.

2006-03-07T19:16:00.000-08:00

Canary Database unapi compliant

It also includes a self-export page:

2006-03-04T19:01:00.000-08:00

Technosophia unapi support

Technosophia by Michael J. Giarlo is also unapi compliant, see his post, and a screenshot below:

2006-03-03T20:53:00.000-08:00

keep list of unapi server (and lanudry list ;-)

Any serious validation should use Ed Summers's excellent validator , although I may still update information in this page (2006-03-08)

while playing with unapi_link script, I notice some unapi site are not 100% unapi compliant, and actually reveal some interesting questions about unapi itself. Before edsu has a test tool running, I need keep a record and to repeat Alan Kent's philosphy "Nothing like airing dirty laundry to get people to clean up their act! :-)". I will try to keep the list up-to-date. A common pitfall list is also created below.

Notice I am doing manual testing, so I apologize for any fault at my side.

1) quaedam
pass

2)http://rsinger.library.gatech.edu/unapi/sru.php
a) a trailing "?" in unapi link. (notice the spec said no trailing ?)
b) request to a dissemination doesn't return right value

3) http://staff.washington.edu/leftwing
pass

4) wikid
client-side xslt render is required. not sure how to handle this, because we cannot expect all client has an xslt rendering ability, I would like to say unapi server must explicitly support HTML page.

4) opa
4.1) doesn't return json in formats request.
4.2) no site formats list, I think unapi should clarify relationship between site and individual formats list. The wording of OAI-PMH is more precise: "If this argument (identifier) is omitted, then the response includes all metadata formats supported by this repository." In this case I think an empty list is correct. In particular, I think unapi can be cleaner by following how to deal with ListMetadataFormats in OAI-PMH.

5) unapi_link
need handle relative path in unapi link.

6)canarydatabase
pass
Notice is also has a page explicitly exposing unAPI links.
canarydatabase export

7) evergreen
it doesn't work with unapi_link, still looking for the reason.

Common pitfalls

1) use href instead of xhref in UNAPI link
2) The trailing "?" is explicitly forbidden in unAPI spec.

Please leave a comment or send me an email if any question

2006-03-02T08:26:00.000-08:00

UTF-8 character ”RIGHT DOUBLE QUOTATION MARK, or ” or ” or UTF-8 e2809d.

This takes me a while staring at the screen and figure out the problem.

In one web page I am testing, the RIGHT DOUBLE QUOTATION MARK (unicode 0x201d); character is used to quote attributes, instead of plain quote (unicode 0x22). the problem is that these character looks terribly similar, and usually in debug tool you don't realize the differences.

Anyhow it takes me hour to figure out the problem, so perhaps better record the steps here.

1) use "od -ax" find utf-8 value of this character e2809d.

2) get the unicode 0x201d by following formula in [http://en.wikipedia.org/wiki/UTF-8] to get its unicode value.

3) search the unicode either in "Character Map" of redhat, or search it in the web [http://www.fileformat.info/search/search.htm]

2006-02-28T22:52:00.000-08:00

pubmed and citeseer also unapi-enabled by opa

It's hard to catch Dan's speed of adding more unapi-enabled sites [http://onebiglibrary.net/project/opa/opa-0.2-release-with-json-wrapper]. I am talking about all these OAI-PMH repositories -- perhaps I can turn nearly obsolete DP9 to unapi-compliant, which will be easier than writing several hundreds greasymonkey script.

Anyhow, I have added citeseer and pubmed, these are very similar to previous amazon script. Citeseer has some trick with their identifiers (e.g. http://citeseer.ist.psu.edu/brin98anatomy.html and http://citeseer.ist.psu.edu/285516 refers same article), but it seems to be working now.

See a screenshot for pubmed below:

and citeseer

To try it out for citeseer:

1. Install greasemonkey, restart firefox, come back here, etc
2. Install citeseer_opa.user.js
3. Install unapi_link.user.js
4. Visit a citeseer page, such as
http://citeseer.ist.psu.edu/285516

Notice the sequence of operation is important.

To try it out for pubmed:

1. Install greasemonkey, restart firefox, come back here, etc
2. Install pubmed_opa.user.js
3. Install unapi_link.user.js
4. Visit a pubmed page, such as
http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=pubmed&dopt=Abstract&list_uids=12345678&query_hl=1&itool=pubmed_docsum

Notice the sequence of operation is important.

2006-02-26T19:24:00.000-08:00

unapi-enabled amazon.com with OPA proxy.

I put together an unapi-enabled amazon page, it uses Dan's OPA-0.1 server as proxy [http://onebiglibrary.net/story/opa-release-0.1].

Again I am following Dan's previous Amazonmixer script [http://curtis.med.yale.edu/dchud/log/project/rogue/amazon-coinspmh-enabled,

See a screenshot below:

To try it out:

1. Install greasemonkey, restart firefox, come back here, etc
2. Install amazon_opa.user.js
3. Install unapi_link.user.js
4. Visit an amazon book page, such as http://www.amazon.com/gp/product/0131103628/104-2542103-6763135

Notice the sequence of operation is important.

so I did nothing new but to demonstrate the system can work ;-) There is a small problem with list of metadata formats supported at a site, since OPA 0.1 returns an empty list for the site. In this version I use listformats response from first available URI. I think we still need a little clarification between listformats for a site and an identifier.

2006-02-25T16:59:00.000-08:00

Make a case for URI microformat

The URI microformat, as suggested in unapi specification by Dan Chud, will be very interesting to library application. Actually it takes advantage of OpenURL and existing link resolver solution.

URI microformat defines a convention of plugging URI metadata in HTML page.


<span class=”unapi-uri” title=”info:pmid/12345678”>PMID 12345678</span>

If URI microformat is adopted and widely used, say in major online bookstore, or in faculty/researcher's homepage, now a microformat-aware application (be a greasemonkey script, or a web service) can grab the identifier and point to your local OpenURL resolver, you immediately get the copy from local library.

In this sense, the URI microformat is very similar to COINS, but it's much simpler and cleaner, anyone can understand and use it, and its aplication can be beyond traditional research library. e.g. in a public library, you can use amazon as catalog and immediately check if it's available in local collection.

Maybe the rosy picture is too opmistic, but I think this thing is something really valuable.

2006-02-24T13:48:00.000-08:00

Thinking about identifier and unAPI

This topic is brought up again in gcs-pcs list, the question is whether to use unapi&id=xxx or unapi&uri=xxx. So many smart people have spent quite some time on the topic of identifier, I won't pretend I know what I am talking about ;-) but just some personal thought.

First I think whether an identifier is persistent: horizontal (time) or vertical (across different applications) is really an economic issue. There are really three categories: precious, normal, or free one.

Sometimes the "thing" is so precious, so people takes extra care, such as DOI, ISBN, handle, PURL, or info URI, another example is the w3c's tech reports always reside in same place. All these needs central control and special care ($$$) are taken.

In second category, we do care but it doesn't worth the extra effort, good examples are such as tag URI, Permanent Link of blog, or various unregistered URIs, people is using them everyday and it works.

The third category is most common URLs, we put it there just because it's resolvable at certain time. There is no guarantee that it will be ever be available tomorrow, and we all live well with this.

My point is that all these identifiers have good reason to exist, and which one to choose is essentially an economic model. And we cannot predict which one will fly and market will tell us.

Now come back to unAPI, I think URI is essentially important because it's cornerstone of the Web, the whole RDF thing is based on URI, we just cannot easily discard it. Second, perhaps weak argument is that using URI will make people think twice before putting something there, therefore help persistence and re-use.

The beauty of URI perhaps can be demonstrated by following example: in blog world people uses "Permanent Link", we can easily plug unapi to blogspace by doing:

unapi?uri=http://www.inkdroid.org/journal/2006/02/24/hit-sh/&format=dc

unapi?uri=http://www.inkdroid.org/journal/2006/02/24/hit-sh/&format=html

This is really cool because we suddenly have access to rich metadata for all web information. Perhaps people will argue that "http://www.inkdroid.org/journal/2006/02/24/hit-sh/" is not permanent -- again, this is an economic issue, and it perhaps is more persistent than handle ;-)

The last thing is about what copy/paste means in unAPI. One camp said we are copying unapi/?uri, andother camp said that we are copying uri.

Although initially perhaps unapi/?uri is feasible, I think the final goal is to be able to copy/paste uri. I guess this is perhaps Dan's original vision: there are really two parts in unapi, a microformat to specify URI; and a mechanism of accessing them. I seriously think the first part is very important and independent.

Now about second part, it's really building a way of specifying possible services to an URI, and responses format of these services. This excites me a lot. We all know REST model, however REST model only specifies request, it doesn't say anything about
response. However, if unAPI is really successful, it actually adds another aspect to REST. So if I am not mistaken, I saw a huge potential of integrating library with the web.

2006-02-24T08:07:00.000-08:00

unapi_link script to add unapi links

Inspired in #code4lib, I am getting interested in Greasemonkey, I started by reading "Dive into Greasemonkey" and studying Dan Chud's COINS-PMH code. Here is a little script of adding unapi links to unapi-compatible page, it adds links right next to unapi span.

To try it out:

Install greasemonkey, restart firefox, come back here, etc
Install unapi_link.user.js right-click (ctrl-click on mac)
Visit a unapi compatible page, such as quaedam

Notice unapi apis links will appear, but I agree it's difficult to find the tiny link ;-), to make the point, I also put a screen caputure here.

2006-02-21T07:56:00.000-08:00

Cornell web library We heard the new before, but recently William Arms have two publications about Cornell's work of web library -- transfer, storage, and access of whole Internet Archive Collection (tens of billions pages with 600+TB data, still counting ;-).

The project is ambitious and they only start doing real testing in January, so results are initial. Nevertheless this is very related to aDORe work for its immense scalability problem. It is very interesting to see their design choices and arguments, such as transfer rate of the data, pre-ingest, one big SQL server, and one big machine to do everything.

http://www.dlib.org/dlib/february06/arms/02arms.html

http://www.infosci.cornell.edu/SIN/WebLib/papers/Arms2006a.doc

2006-02-19T21:19:00.000-08:00

switch esc to alt key for emacs in macOS.

I was trying to make emacs work as I am comfortable in linux. One issue is to use "alt" key intead of "esc" key for "meta" character.

This is done by "defaults write com.apple.x11 swap_alt_meta -boolean true" in shell script

2006-02-19T20:49:00.000-08:00

replace ^M (\n\r) with \n in emacs

While I am copying firefox text to an emacs buffer, I sometimes get the annoying "^M". This seems solvable by : "M-x replace-string ^q^m RET ^q^j",

2004-11-04T08:09:00.000-08:00

count speed in log filesI by simply average over the last 10 minutes

# echo `tail -35000 netricsd.log| grep -c 03/Nov/2004-18:4` 5k 600 / p | dc

2004-11-04T08:08:00.000-08:00

bc and dc

http://www.computerhope.com/unix/udc.htm
http://www.computerhope.com/unix/ubc.htm

2004-09-09T10:13:00.000-07:00

slope/cofficient of power law distribution

coef(lm (log10(vouty) ~ log10(voutx)))? This the
slope / coefficient of log10(voutx) should be an estimate of the power.

http://maths.newcastle.edu.au/~rking/R/help/03a/7022.html

2004-09-08T10:16:00.000-07:00

Pernbeger's (2004) finding that download counts (what we call "usage
impact") of British Medical Journal articles predict citation
counts ("citation impact") for those articles in subsequent
years confirms what Tim Brody's online usage/citation correlator
http://citebase.eprints.org/analysis/correlation.php has been
demonstrating for several years now across a number of areas in physics
and mathematics (Brody & Harnad 2004, in prep.): There is a significant
correlation between downloads today and citations two years later.

This correlation has two immediate implications:

(1) Download counts can be used as early performance indicators for
papers and authors, even before their impact is reflected in citation
counts: http://citebase.eprints.org/ ( Hitchcock et al. 2003).

(2) Enhancing usage impact is yet another reason for authors to
provide open access to their articles by self-archiving them.

References

Brody, T. & Harnad, S. (2004, in prep.) Using Web Statistics as a
predictor of Citation Impact.
http://www.ecs.soton.ac.uk/~harnad/Temp/timcorr.doc

Harnad, S. & Brody, T. (2004) Comparing the Impact of Open Access (OA)
vs. Non-OA Articles in the Same Journals, D-Lib Magazine 10 (6) June
http://www.dlib.org/dlib/june04/harnad/06harnad.html

Hitchcock, Steve; Woukeu, Arouna; Brody, Tim; Carr, Les; Hall, Wendy and
Harnad, Stevan. (2003) Evaluating Citebase, an open access Web-based

2004-07-26T09:49:00.000-07:00

Python, Zope, and Plone
http://www.plone.org/
http://www.zope.org

Zope is an application server and Plone is a content management tool over Zope.

Sounds interesting and gets a lot of attention.

2004-07-13T14:06:00.000-07:00

Tomcat 5 with arc package

in tomcat 5.0.25, it appeared that tomcat 5 disable the invoker servlet
by default. You may read the details from
http://www.coreservlets.com/Apache-Tomcat-Tutorial/#Enable-Invoker

The quick solution is to uncomment the corresponding servlet element at
the top of install_dir/conf/web.xml, as below:

invoker

org.apache.catalina.servlets.InvokerServlet

...

2004-07-13T14:01:00.000-07:00

Static and Dynamic table in mysql

http://dev.mysql.com/doc/mysql/en/Static_format.html
http://dev.mysql.com/doc/mysql/en/Dynamic_format.html

Static format is the default for MyISAM tables. It is used when the table contains no variable-length columns (VARCHAR, BLOB, or TEXT). Each row is stored using a fixed number of bytes.

Dynamic storage format is used if a MyISAM table contains any variable-length columns (VARCHAR, BLOB, or TEXT), or if the table was created with the ROW_FORMAT=DYNAMIC option.

2004-07-13T13:57:00.000-07:00

TigerHash

Tiger is a fast new hash function, designed to be very fast on modern computers, and in particular on the state-of-the-art 64-bit computers (like DEC-Alpha), while it is still not slower than other suggested hash functions on 32-bit machines.

http://www.cs.technion.ac.il/~biham/Reports/Tiger/

The hash length 192 bits. Accordingly, the hash length of MD5 is 128 bits, and SHA-1 is 190 bits.

2003-08-14T09:21:00.000-07:00

Improve performance of tomcat/cocoon
overwrite doGet in servlet

doHead{

protected void doHead(HttpServletRequest req,
HttpServletResponse resp)
throws ServletException,
java.io.IOException

Receives an HTTP HEAD request from the protected service method and handles the request. The client sends a HEAD request when it wants to see only the headers of a response, such as Content-Type or Content-Length. The HTTP HEAD method counts the output bytes in the response to set the Content-Length header accurately.

If you override this method, you can avoid computing the response body and just set the response headers directly to improve performance. Make sure that the doHead method you write is both safe and idempotent (that is, protects itself from being called multiple times for one HTTP HEAD request).

If the HTTP HEAD request is incorrectly formatted, doHead returns an HTTP "Bad Request" message.
}

2003-08-12T13:39:00.000-07:00

Compression and de-compression with Java:

http://developer.java.sun.com/developer/technicalArticles/Programming/compression/

Testing code available at project directory.

2003-08-12T13:22:00.000-07:00

Googlesets:

http://labs.google.com/sets

Is this an application of spreading activation?

2003-08-12T13:21:00.000-07:00

Cocoon/Tomcat

Cocoon/Tomcat silently discards concurrent requests if the request queue is full. This can solved by changing server.xml
port="8080" minProcessors="5" maxProcessors="75"
enableLookups="true" redirectPort="8443"
acceptCount="10" debug="0" connectionTimeout="60000"/>

2003-07-24T14:48:00.000-07:00

Other Useful packages recently discovered.
octave
http://www.octave.org/
Open Source Matlab

Gnuplot
great drawing tool
http://www.gnuplot.info/

Boost, previous Generic Graph Component Library (GGCL), looks like a powerful c++ graph package

PathView

Great 3-D visulization, if we want to build a real cool galaxy of citation network ;-)
http://www.haydenplanetarium.org/hp/vo/partiview/

R
http://www.r-project.org/
statistics package with a good social network analysis module

2003-07-24T14:42:00.000-07:00

install webdot/graphviz/tomcat/apache at intrepid:/home/liu_x/tool
Explore Java matrix/Graph packages

General Java matrix/math packages:
http://math.nist.gov/javanumerics/

It has many useful links to other java math projects.

Colt package:
It looks powerful but I did not find a graph theory API
http://hoschek.home.cern.ch/hoschek/colt/index.htm

JDSL:
http://www.cs.brown.edu/cgc/jdsl/doc.html
It has limited graph algo implementation, including DFS, DijkstraPath. It's more like a teaching tool.

Graph Foundation Class for Java
http://www.alphaworks.ibm.com/tech/gfc
It has a more powerful Graph API, however the latest version is released 07/31/2000. There is no much activity going on there. And the interface is rather odd. It supports breath-first search though.

2003-07-24T14:23:00.000-07:00

Install Tomcat 4.1.24/Apache 2.0/jk2/GraphViz, many documents are incomplete or not work at all

Here are two useful links
http://www.greenfieldresearch.ca/technical/jk2_config.html
http://www.pixelfreak.net/howto/apache2_jk2_tomcat/socket.html

Steps:
--Compile Apache with so enabled
./configure --prefix=/usr/local/apache2 --enable-ssl --enable-module=so

--Install Tomcat
Downlod jk2 and compile
cd to /usr/local/src/jk2/jakarta-tomcat-connectors-jk2-2.0.2-src/jk/native2 and run configure with the following options...
./configure --with-apxs2=/bin/apxs --with-tomcat41= --with-java-home= --with-jni --with-pcre

--copy .so files to apache modules directory

--Config httpd.conf
LoadModule jk2_module modules/mod_jk2.so

--create workers.properties at $apache/conf
Define the communication channel
[shm]
file=/home/liu_x/tool/apache2/logs/jk2.shm
[channel.socket:localhost:8009]
info=Ajp13 forwarding over socket
tomcatId=localhost:8009

[ajp13:localhost:8009]
channel=channel.socket:localhost:8009

# Map the Tomcat examples webapp to the Web server uri space
[uri:/examples/*]
worker=ajp13:localhost:8009

--create jk2.properties at $tomcat/conf

shm.file=/home/liu_x/tool/apache2/logs/jk2.shm

-- Start tomcat first, and then start Apache.

2003-02-17T19:13:00.000-08:00

good day

2003-02-17T19:10:00.000-08:00

Today I took cs772 courses