DAS registry installation notes

From BioDAS
Jump to: navigation, search

Gregg Helt's notes on getting Sanger DAS registry running locally (on OSX)

Java code:

BioJava: checked out head of trunk (svn://code.open-bio.org/biojava/biojava-live/trunk)
Dasobert: checked out head of trunk (http://www.derkholm.net/svn/repos/dasobert/trunk)
DasRegistry: checked out head of openid branch, as per disccusion with Jonathon 
    To build dasregistry, started new NetBeans project as "Web Application with Existing Sources"
                rather than free-form project, because having problems with checked in build.xml file
                this way, NetBeans creates a new build file (nbbuild.xml)

JARs required but not present in repository:
mysql-connector-java-5.1.7-bin.jar (MySQL Connector/J JDBC driver)
axis.jar (Axis 1.4)
biojava.jar (head of biojava trunk)
dasobert.jar (head of dasobert trunk)
commons-logging-1.0.4.jar (Apache commons)
mail.jar (javamail-1.41)
ols-client.jar (Ontology Lookup Service)
commons-codec-1.3.jar (Apache commons)
commons-httpclient-3.0.1.jar (Apache commons)
commons-logging-1.03.jar (Apache commons)
htmlparser.jar (from openid-sxip)
openxri-client.jar (from openid-sxip)
openxri-syntax.jar (from openid-sxip)
log4j-1.2.15.jar (Apache log4j)
dom3-xercesImpl.jar (from openid-sxip endorsed libs)
dom3-xml-apis.jar (from openid-sxip endorsed libs)
xalan-2.6.0.jar (from openid-sxip endorsed libs)
commons-fileupload-1.2.1.jar (Apache commons)
jcommon-1.0.13.jar (from jfreechart)

Code changes required for registry compilation:

Couldn't find ontology lookup jar that corresponded with naming in head of registry oid branch: The only compilation problem I had was with the DasRegistryOntologyLookup class, which depends on classes in the uk.ac.ebi.www.ontology_lookup.OntologyQuery package. Which I couldn't locate. The closest I could find was the Ontology Lookup Service code at http://www.ebi.ac.uk/ontology-lookup/. As near as I can tell this has the exact same classes as uk.ac.ebi.www.ontology_lookup.OntologyQuery, except they're in a different package: uk.ac.ebi.ook.web.services. So I'm guessing there was just a package refactoring at some point and the registry is using either a newer or older version of the ontology-lookup code than what is available at the OLS site. Changed the imports in DasRegistryOntologyLookup to use the package names in the OLS jar and now everything compiles. Should check with Jonathon and see if where to find ontology_lookup.OntologyQuery package, and whether I should swap in.

Changes required to avoid runtime errors:
        org.biojava.services.das.dao.DasSourceManager.getCapabilities() method
        had to add "sources" column to SELECT statement, otherwise get "column 'sources' not found" errors when accessing ResultSet

    Tomcat / webapp setup:
    JNDI stuff in Tomcat/webapp couldn't find mysql JDBC drivers even when they
        were added as project library jars, and in several places under Tomcat $CATALINA_HOME
    had to add a RESOURCE element to context.xml in project Configuration Files:
       <Resource name="jdbc/mysql" auth="Container" type="javax.sql.DataSource"
               maxActive="8" maxIdle="4" maxWait="10000"
               username="javauser" password="javadude" driverClassName="com.mysql.jdbc.Driver"
      (see ~/release/install.txt for Andreas' recommendation on a slightly different Resource declaration)

    Servlet package naming in WebContent/WEB-INF/web.xml is out of sync with source code, so fixed in web.xml:
    (all are just changing "servlet" to "servlets" in servlet-class element class names
    org.biojava.services.das.servlet.Das2RegistryServlet --> org.biojava.services.das.servlets.Das2RegistryServlet
    org.biojava.services.das.servlet.CoordSysServlet --> org.biojava.services.das.servlets.CoordSysServlet
    org.biojava.services.das.servlet.FileUploadServlet --> org.biojava.services.das.servlets.FileUploadServlet
    org.biojava.services.das.servlet.DasSourceActionServlet --> org.biojava.services.das.servlets.DasSourceActionServlet
    org.biojava.services.das.servlet.ProjectIconServlet --> org.biojava.services.das.servlets.ProjectIconServlet
    org.biojava.services.das.servlet.ProjectActionServlet --> org.biojava.services.das.servlets.ProjectActionServlet

WARNING!! Xerces parser problems (org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found) conflicts between JDK1.5 internal Xerces (crimson?) and explicit setting of implementations in org.biojava.services.das.registry.RegistryConfiguration (also explicitly set often in dasobert examples, Das1Validator, some .jsp files, and tests)

Problem is that internal Xerces is auto-overriding any external Xerces (in my case from dom3-xercesImpl.jar) in classpath but with different package naming, which gives errors like:

javax.xml.parsers.FactoryConfigurationError: Provider org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found

See http://www.mbaworld.com/docs/class-loader-howto.html , section on XML parser class loading and

http://java.sun.com/j2se/1.5.0/docs/guide/standards/index.html for endorsed libs mechanism for overriding internal JDK/JRE classes

Unfortunately can't just use endorsed libs in a straightforward ways because there is also a problem with NetBeans using anything other than the JDK1.5 internal lib, which it seems to interpret as having Xalan in the classpath, which causes NetBeans to refuse to run at all: http://wiki.netbeans.org/FaqXalanOnCPmplementations

Tried several ways of using endorsed libs mechanism in combination with using different JDK to run NetBeans (to avoid Xalan problem mentioned above), but never could get the combination to stick on NetBeans6.5 under OSX -- possibly the problem is in specifying the java.endorsed.dirs system property?

SOLUTION: for now, commented out explicit setting of factory in RegistryConfiguration


Registry MySQL database:

In general, preface everything on command-line with sudo in order to admin stuff... Alternatively, for mysql command line login as root:

       mysql -u root -p

Make sure MySQL is installed

Starting up MySQL daemon (as per below):

   sudo /usr/local/mysql/bin/mysqld_safe asks for password
   [Cnt-Z to stop process]
   bg to restart process in background

Set password for root, admin, to standard internal via command-line:

   sudo mysqladmin -u root password ********

Create das2registry01 database:

   sudo /usr/local/mysql/bin/mysqladmin create dasregistry01
Populated database from registry SQL file that AndyJ sent me
    (which I think is a mysqldump of the current registry database,
         file is at ~/projects/external/das/das_registry_oid_branch/manually_added_db/reg.sql)         

sudo mysql -p dasregistry01 < reg.sql

Added user: "javauser", password: "javadude" via mysql client:

   -> grant all privileges
   ->  on *.* to 'javauser'@'localhost'
   ->  identified by 'javadude'
   ->  with grant option;

first had to set root/admin password, otherwise got error:

  "Access denied for user @'localhost' (using password: NO)"

see discussion at ??? on problem with using MySQL JDBC drivers without password

To see users:

  select host, user, password from mysql.user

Use DbVisualizer application to browse database:

   current connection alias: "local MySQL"
   database URL: jdbc:mysql://localhost:3306/
        (or optionally to see _just_ dasregistry01 database:
               jdbc:mysql://localhost:3306/dasregistry01 )
   userid: root (or optionally "javauser")
   password:  ****** (or optionally "javadude")

Older installation notes in code repository

Only after figuring out all of the above did I find Andreas' notes from Nov 2005 on local installation of the DAS registry, in the code repository at
Also the SQL for creating an empty database registry:
And the (old and out of date) war file for easy distribution
WARNING!!!  The installation notes, SQL file, and .war are out of date,
    at least for the openid branch --
         for example:
            the SQL does not include creation of "sources" and "interaction" columns in the "registry" tab
            the .war does not include _anything_ from Dasobert
There are additional jars in $CHECKOUT_ROOT/release/version2.0/ that date from July 2006
        biojavar.jar, dasobert.jar, dasregistry.jar
        These are closer to current code base, for example there are Das2Source etc. classes which were not present in Nov2005 war
        But still out of date -- for instance nothing on molecular interactions in dasobert jar
STILL NEED TO DO: the installation notes suggest putting the RESOURCE configuration for jdbc/mysql in the $TOMCAT/conf/server.xml config file instead of in the webapp context.xml file as I did above

Remaining Issues

Still getting warning messages about log4j from Tomcat when registry webapp is deployed: log4j:WARN No appenders could be found for logger (org.springframework.context.support.ClassPathXmlApplicationContext). log4j:WARN Please initialize the log4j system properly.

Problems with das.xsl stylesheet

    Browser is having trouble retrieving the das.xsl stylesheet
    Because of the way the webapp is configured (mapping das1/*) and the way the stylesheet is specified in the sources XML (just as "das.xsl"), the browser does a GET on [rootpath]/das1/das.xsl.  The server, rather than directly returning the das.xsl file for this request, hands off responsibility to the Das2RegistryServlet.  Which retrieves the file as a resource stream from the servlet context and then streams it back as content of the HTTP response.  But resource-not-found exceptions are getting thrown.  Looks like problem is that at least some combinations of JVM/TomCat really want a "/" prefixed to the file name in arg to the ServletContext.getResourceAsStream() call (see http://www.xyzws.com/servletfaq/how-to-use-servletcontextgetresourceasstream/18)

SOLUTION: added conditional so that if "das.xsl" doesn't work for resource retrieval, add a "/" prefix and try again.

Also repository is missing a bunch of css stylesheet files, referred to in the das.xsl (and therefore like das.xsl, they are triggering attempts to load as resources in Das2RegistryServlet calls (in response to browser GETs).


TEMPORARY SOLUTION: turned off adding of xml-stylesheet clause to sources XML response in ServletResponseWriter (made adding it a boolean option)

NEED TO FIX: once css is checked into repository, can flip boolean to turn xml-stylesheet back on)

Looking at adding support for DAS2 capabilities

Starting to modify to support DAS2 "segments", "types", "features" capabilities (and extend "sources" support)

Added columns to registry database for das2:segments, das2:types, das2:features mysql -u root -p

 > use dasregistry01;
 > alter table registry add das2_segments bit;
 > alter table registry add das2_types bit;
 > alter table registry add das2_features bit;


     added constants/enums for DAS2 segments, types, features capabilities


     changed how get_capabilities() database query is constructed, so it is based on capabilities specified in org.biojava.dasobert.das.Capabilities constants, rather than having to be manually synchronized with Capabilities.  So addition of DAS2 capabilities constants to Capabilities is automatically reflected in DasSourceManager.  This also fixes previous runtime error problem with "sources" absence in database query, but in a more general way than original fix mentioned above.

Major problem going forward is that the registry database doesn't store capability URLs, but rather a URL for the versioned source and a set of BIT/boolean columns for whether a versioned source supports a particular capability. As near as I can tell the query_uri of the capability is then composed from the versioned source URI and the column name corresponding to a capability.

It looks like there was work done by Andreas on a revised registry database schema that supported decoupling of capability URIs from versioned source URIs via source, capability, and source2capability tables. Is there an instance of this database still around?