|Baubles: Making the World a Better Place for Logfile Viewing||CCP4 Newsletter 47 Winter 2007|
Peter Briggs* and Kevin Cowtan**
*CCP4, CSE Department, STFC Daresbury Laboratory, Warrington WA4 4AD
**Structural Biology Laboratory, University of York, Heslington, York
Email: cowtan at ysbl.york.ac.uk
It is a perennial problem with running many CCP4 programs, that in order to understand the results of a program run it is often necessary to look at the log file from that program and this can sometimes be quite challenging. Traditional methods such as viewing the "raw" log file using the UNIX/Linux more command in a terminal window (see figure 1a) have given way to tools such as the file viewer and loggraph programs in CCP4i (figure 1b). However the log file can still be difficult to navigate, particularly if it consists of the output from many programs run in sequence - as is often the case for CCP4i tasks.
|Figure 1a) Log file viewed using more in a Linux terminal window.||Figure 1b) Log file displayed using CCP4i viewer, with graphs shown in separate loggraph window.||Figure 1c) Log file reformatted by baubles and displayed in web browser with inline graphs in JLogGraph.|
Baubles is an experimental program that attempts to make log file navigation and interpretation easier, by reformatting them into HTML so they can be viewed in a web browser (figure 1c).
Baubles is written in Python and uses the smartie module to do the work of interpreting log files (see the article in newsletter 46: http://www.ccp4.ac.uk/newsletters/newsletter46/articles/smartie.html). It runs on Linux and Windows systems and the generated output should be viewable using both Firefox and Internet Explorer.
The best way to get a feel for what baubles does is to look at a few examples:
The Scala and Buccaneer examples each include a "results" section, which is transformed by baubles and placed in a prominent position at the start of each program summary. Similarly the Truncate example includes some warning messages that have been extracted and displayed prominently by baubles.
Some other examples can also be found at http://www.ccp4.ac.uk/peter/logfiles/.
Baubles is still very much an experiment and there are a number of limitations, for example:
You can give baubles a try now - a version of baubles is available for download from:
This file includes an installer script that will update CCP4 6.0.2 to make baubles available from CCP4i (sorry it only installs into Linux and UNIX-type systems at the moment - see the README file for instructions on how to manually install the update for Windows).
Once baubles has been installed, any new jobs that are run in CCP4i will also run baubles at the end to generate an "annotated" version of the log file which can then be accessed via a "View Annotated Log in Web Browser" option in the "View Files from Job" menu (see figure 2).
You can make your program output more "baubles-compatible" by following a few guidelines:
Use the standard CCP4 banners to flag the start (and end, if you're adventurous) of the output from your program - cut and paste from an existing program (but be careful not to add extra #'s!).
Use the $TABLE markup for tables of data that should be displayed as graphs.
Write summary tags <!-SUMMARY_BEGIN--> and <!--SUMMARY_END--> into your program output to enclose any sections that should be particularly interesting to users, for example the key results at the end of the program run.
Use the $TEXT markup for warnings or informational messages. For example, warning messages can be flagged using:
$TEXT:Warning: $$ Stuff here is ignored $$ Something very important that the user really ought to know about! $$
while a "result" section can be denoted by:
$TEXT:Result: $$ Stuff here is ignored $$ Put details of the major results here. $$
(Note that there should only be a single "result" section per program log). In both cases, the messages will be extracted and displayed prominently within the baubles-annotated log file. In the case of the "results" section, additional "magic" formatting is applied in baubles to make the information more readable (for example by putting tabulated data into a HTML table).
Information on the $TABLE and $TEXT markup formats can be found in the CCP4 documentation, for example at http://www.ccp4.ac.uk/dist/html/loggraphformat.html. In addition, the article on smartie in the previous newsletter has some information about the kind of log file features that baubles is able to recognise. You can also email us and we'll try to help make your program output more baubles-compliant and (hopefully) more user-friendly.
Give baubles a try on your favourite programs, and send feedback to us about how we might improve it. Also send feedback to program developers suggesting which bits of program log file you would like to see in a summary.
Baubles is still quite experimental, however we intend to incorporate it in some form in the next release of CCP4. Beyond that we hope to look at ways of improving baubles reformatting to make it more useful operating on "standard" logfiles, and at the same time to put pressure on program developers to make improvements to the "raw" log files output from their programs. Beyond that we would also like to investigate ways to link to other relevant resources and possibly include "commentaries" on the program output.
Baubles was originally written by Peter Briggs and is based on the "smartie" Python module. It has since undergone several iterations of development by both Peter Briggs and Kevin Cowtan. Kevin also produced the original changes to CCP4i to integrate baubles into it, along with the installer script. Both have subsequently been adapted by Peter.
The JLogGraph Java applet was written by Kevin Cowtan who has also made several recent substantial improvements.
Phil Evans has provided some invaluable feedback on both baubles and JLogGraph, for which PJB and KC are both very grateful.