M.L.K. Newspaper comparison using JRAPublish

M.L.K. Newspaper comparison using JRAPublish

a study by PlanetDjVu, February 12, 2004

Overview

This study is inspired by both the recent release of JRAPublish 2.0 and a return visit to the NewspaperArchive.com website. NewspaperArchive.com was an early adopter of the DjVu format for the conversion of newspaper microfilm to digital web documents, but a year or two ago, for reasons not explained (perhaps a "political" decision), they switched to PDF using CCITT Group 4 compression. One thing we can say for sure is that this decision was not made for the purpose of delivering content faster to the end user, because the current format and compression method are much slower, as we demonstrate.

NewspaperArchive.com has published 6 sample newspaper pages paying tribute to Martin Luther King, which can be viewed here: http://martinlutherkingjr.newspaperarchive.com/

Study Method

We downloaded the 5 PDF newspaper pages and then used them as input files in JRAPublish. We generated dual searchable-image DjVu and PDF files, applying the JBIG2 compression method to the PDF files. It was necessary to use the "force pages to bitonal" method for processing the input PDF pages, because even though the page images were bitonal, there was unnecessary colorspace information stored in the PDF. This is an all-too-frequent problem with bitonal-mage PDF files created by other software.

Results

Click on any size in the chart below to open that file. Compare the time it takes to open each version. You will see that by far the fastest performers are the DjVu files, followed by the JBIG2-compressed PDF files generated by JRAPublish 2.0.

Featured Article Title	G4-PDF	JBIG2-PDF	DjVu
Manhunt On for King's Assassin; Nation Grieves	539 Kb	337 Kb	258 Kb
Violence follows King's Death	853 Kb	540 Kb	362 Kb
King to Ignore Ban on March	642 Kb	451 Kb	318 Kb
New Looting, U.S. Moves More Troops Into Capital	497 Kb	345 Kb	237 Kb
Taft Urges King to Cancel March	445 Kb	316 Kb	249 Kb
Warning Signals Fly	638 Kb	443 Kb	259 Kb
Total	3.614 Mb	2.432 Mb	1.683 Mb
Percentage	100 %	67 %	47 %

OCR Comparison

The OCR layer of the PDF from NewspaperArchive.com has more recognition errors than the OCR layer of JRAPublish. Also, there is a problem of words running together in NewspaperArchive.com, and the highlight rectangles to not map correctly over the text when you select it. In PDFand DjVu files produced by JRAPublish, the highlight rectangles are very accurate.

In the second table, we show the results of OCR from Document Express 4.1 and Adobe Acrobat 6.01 Capture Plug-in. Both results have significant errors that do not occur in the OCRed text generated by JRAPublish.

Image of text	OCRed Text generated by JRAPublish	OCRed Text from NewspaperArchive.com
	MEMPHIS, Tenn. (AP) � Authorities pressed a manhunt today for the killer of Dr. Martin Luther King Jr. whose assassination yesterday touched off Negro violence in a number of American cities and brought a national out pouring of grief and sorrow. King. 39, leading advocate of nonviolence and Nobel Prize winner, died in a Mem phis hospital last night less than an hour after he was shot in the neck by a white gun man while standing on the balcony of his motel. Police director Frank Hollo- man said today that a single white man, following an ap parently well planned proce dure, was the assassin.	MEMPHIS, Tenn. (AP) ??? Authoritiespressed a manhunt todayfor the killer of Dr. Martin Luther KingJr. whose assassination yesterday touchedoff Negro violence in a number of American cities and brought a national outpouring of grief and sorrow. King. 39, leadingadvocate of nonviolence and Nobel Prize winner, died in a Memphis hospital last night less than an hour after he was shot in the neck bya whit* gunman while standing on the balcony of hismotel. PolicedirectorFrank Holloman said todaythat a single white man, following an apparently well planned procedure, was the assassin.

Image of text	OCRed Text generated by Document Express	OCRed Text generated by Adobe Acrobat Capture
	MEMPHIS, Tenn. tAP) -- Authorities pressed a manhunt today for the killer of Dr. Martin Luther King Jr. whose assassination y e s t e r d a y touched off Negro violence in a number of American cites and brought a national out- pouring o( grief and sorrow. King. , leading advocate of nonviolence and N o b e 1 Prize winner, died in a Mem- phis hospital last night less than an hour after he was shot in the neck by a whit gun- Dog ay..have o. m balcony of his motel. Police director Frank Hollo- man said today that a single white man, following an ap. y'a- ngs parenfly well planned proee- seen s dure, was the assassin.	MEMPHIS, Tenn. (AP? - Authorities pressed a manhunt today for the killer of Dr. Martin Luther King Jr. whose assassination y e s t e r d a y touched off Negro violence in a number of American cities and brought a national outpouring r$ grief and sorrow. King. 3B. leading advocate of nonviolence and N o b e 1 Prize winner, died in a Memphis hospital last night less than an hour after he was shot in the neck by a white gunman while standing on the balcony of his motel. Police director Frank H o b man said today that a single white man, following an ag psrently well planned procedure, %as the assassin.

Conclusion

The conversion of microfilm to bitonal image files (TIFF, G4 compressed), and then the conversion of these files to searchable-image web documents is sound, but care must be taken to keep file size to a minimum, while maximizing the quality of the recognized and searchable text. JRAPublish 2.0 excels at this task, producing higher-quality files than those generated and used in the commercial NewspaperArchive.com portal. The JBIG2-PDF files encoded by JRAPublish are a big improvement, but for best results use DjVu, or use the dual-format generating capability of JRAPublish to make both file types, and give the choice of file type to the end user!