Difference between revisions of "DAS registry installation notes"

From BioDAS
Jump to: navigation, search
Line 74: Line 74:
 
conflicts between JDK1.5 internal Xerces (crimson?) and explicit setting of implementations in org.biojava.services.das.registry.RegistryConfiguration
 
conflicts between JDK1.5 internal Xerces (crimson?) and explicit setting of implementations in org.biojava.services.das.registry.RegistryConfiguration
 
(also explicitly set often in dasobert examples, Das1Validator, some .jsp files, and tests)
 
(also explicitly set often in dasobert examples, Das1Validator, some .jsp files, and tests)
Problem is that internal Xerces is auto-overriding any external Xerces (in my case from dom3-xercesImpl.jar) in classpath but with different package naming, which gives errors like:
+
 
javax.xml.parsers.FactoryConfigurationError: Provider org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found
+
Problem is that internal Xerces is auto-overriding any external Xerces (in my case from dom3-xercesImpl.jar) in classpath but with different package naming, which gives errors like:  
See http://www.mbaworld.com/docs/class-loader-howto.html , section on XML parser class loading
+
 
    and http://java.sun.com/j2se/1.5.0/docs/guide/standards/index.html for endorsed libs mechanism for overriding internal JDK/JRE classes
+
javax.xml.parsers.FactoryConfigurationError: Provider org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found  
Unfortunately can't just use endorsed libs in a straightforward ways because there is also a problem with NetBeans using anything other than the JDK1.5 internal lib, which it seems to interpret as having Xalan in the classpath, which causes NetBeans to refuse to run at all: http://wiki.netbeans.org/FaqXalanOnCPmplementations
+
 
 +
See http://www.mbaworld.com/docs/class-loader-howto.html , section on XML parser class loading and
 +
 
 +
http://java.sun.com/j2se/1.5.0/docs/guide/standards/index.html for endorsed libs mechanism for overriding internal JDK/JRE classes
 +
 
 +
Unfortunately can't just use endorsed libs in a straightforward ways because there is also a problem with NetBeans using anything other than the JDK1.5 internal lib, which it seems to interpret as having Xalan in the classpath, which causes NetBeans to refuse to run at all: http://wiki.netbeans.org/FaqXalanOnCPmplementations  
 +
 
 
Tried several ways of using endorsed libs mechanism in combination with using different JDK to run NetBeans (to avoid Xalan problem mentioned above), but never could get the combination to stick on NetBeans6.5 under OSX -- possibly the problem is in specifying the java.endorsed.dirs system property?
 
Tried several ways of using endorsed libs mechanism in combination with using different JDK to run NetBeans (to avoid Xalan problem mentioned above), but never could get the combination to stick on NetBeans6.5 under OSX -- possibly the problem is in specifying the java.endorsed.dirs system property?
 +
 
SOLUTION: for now, commented out explicit setting of factory in RegistryConfiguration
 
SOLUTION: for now, commented out explicit setting of factory in RegistryConfiguration
 +
 
NEED TO EITHER COME UP WITH ALTERNATIVE, OR ALSO FIX IN OTHER CLASSES
 
NEED TO EITHER COME UP WITH ALTERNATIVE, OR ALSO FIX IN OTHER CLASSES
  
  
 
'''Registry MySQL database:'''
 
'''Registry MySQL database:'''
 +
 
In general, preface everything on command-line with sudo in order to admin stuff...
 
In general, preface everything on command-line with sudo in order to admin stuff...
 
Alternatively, for mysql command line login as root:
 
Alternatively, for mysql command line login as root:
Line 129: Line 138:
  
 
'''Older installation notes in code repository'''
 
'''Older installation notes in code repository'''
 +
<pre>
 
Only after figuring out all of the above did I find Andreas' notes from Nov 2005 on local installation of the DAS registry, in the code repository at
 
Only after figuring out all of the above did I find Andreas' notes from Nov 2005 on local installation of the DAS registry, in the code repository at
 
     $CHECKOUT_ROOT/release/install.txt
 
     $CHECKOUT_ROOT/release/install.txt
Line 146: Line 156:
 
THIS STUFF NEEDS UPDATING!!!
 
THIS STUFF NEEDS UPDATING!!!
 
STILL NEED TO DO: the installation notes suggest putting the RESOURCE configuration for jdbc/mysql in the $TOMCAT/conf/server.xml config file instead of in the webapp context.xml file as I did above
 
STILL NEED TO DO: the installation notes suggest putting the RESOURCE configuration for jdbc/mysql in the $TOMCAT/conf/server.xml config file instead of in the webapp context.xml file as I did above
 +
</pre>
  
 
'''Remaining Issues'''
 
'''Remaining Issues'''
 +
 
Still getting warning messages about log4j from Tomcat when registry webapp is deployed:
 
Still getting warning messages about log4j from Tomcat when registry webapp is deployed:
 
log4j:WARN No appenders could be found for logger (org.springframework.context.support.ClassPathXmlApplicationContext).
 
log4j:WARN No appenders could be found for logger (org.springframework.context.support.ClassPathXmlApplicationContext).
Line 153: Line 165:
  
 
Problems with das.xsl stylesheet
 
Problems with das.xsl stylesheet
 +
 
     Browser is having trouble retrieving the das.xsl stylesheet
 
     Browser is having trouble retrieving the das.xsl stylesheet
 +
 
     Because of the way the webapp is configured (mapping das1/*) and the way the stylesheet is specified in the sources XML (just as "das.xsl"), the browser does a GET on [rootpath]/das1/das.xsl.  The server, rather than directly returning the das.xsl file for this request, hands off responsibility to the Das2RegistryServlet.  Which retrieves the file as a resource stream from the servlet context and then streams it back as content of the HTTP response.  But resource-not-found exceptions are getting thrown.  Looks like problem is that at least some combinations of JVM/TomCat really want a "/" prefixed to the file name in arg to the ServletContext.getResourceAsStream() call (see http://www.xyzws.com/servletfaq/how-to-use-servletcontextgetresourceasstream/18)
 
     Because of the way the webapp is configured (mapping das1/*) and the way the stylesheet is specified in the sources XML (just as "das.xsl"), the browser does a GET on [rootpath]/das1/das.xsl.  The server, rather than directly returning the das.xsl file for this request, hands off responsibility to the Das2RegistryServlet.  Which retrieves the file as a resource stream from the servlet context and then streams it back as content of the HTTP response.  But resource-not-found exceptions are getting thrown.  Looks like problem is that at least some combinations of JVM/TomCat really want a "/" prefixed to the file name in arg to the ServletContext.getResourceAsStream() call (see http://www.xyzws.com/servletfaq/how-to-use-servletcontextgetresourceasstream/18)
 +
 
SOLUTION: added conditional so that if "das.xsl" doesn't work for resource retrieval, add a "/" prefix and try again.
 
SOLUTION: added conditional so that if "das.xsl" doesn't work for resource retrieval, add a "/" prefix and try again.
  
 
Also repository is missing a bunch of css stylesheet files, referred to in the das.xsl (and therefore like das.xsl, they are triggering attempts to load as resources in Das2RegistryServlet calls (in response to browser GETs).
 
Also repository is missing a bunch of css stylesheet files, referred to in the das.xsl (and therefore like das.xsl, they are triggering attempts to load as resources in Das2RegistryServlet calls (in response to browser GETs).
 +
 
     /css/dazzle.css
 
     /css/dazzle.css
 
     /css/content.css
 
     /css/content.css
Line 163: Line 179:
 
     /css/printer-styles.css
 
     /css/printer-styles.css
 
     /css/screen-styles.css
 
     /css/screen-styles.css
 +
 
TEMPORARY SOLUTION: turned off adding of xml-stylesheet clause to sources XML response in ServletResponseWriter (made adding it a boolean option)
 
TEMPORARY SOLUTION: turned off adding of xml-stylesheet clause to sources XML response in ServletResponseWriter (made adding it a boolean option)
 +
 
NEED TO FIX: once css is checked into repository, can flip boolean to turn xml-stylesheet back on)
 
NEED TO FIX: once css is checked into repository, can flip boolean to turn xml-stylesheet back on)
  
     
+
'''Looking at adding support for DAS2 capabilities'''
Now starting to modify to support DAS2 "segments", "types", "features" capabilities
+
     
(and extend "sources" support)
+
Starting to modify to support DAS2 "segments", "types", "features" capabilities
 +
(and extend "sources" support)  
  
 
Added columns to registry database for das2:segments, das2:types, das2:features
 
Added columns to registry database for das2:segments, das2:types, das2:features

Revision as of 08:36, 16 March 2009

Notes on getting Sanger DAS registry running locally (on OSX)

Java code:

BioJava: checked out head of trunk (svn://code.open-bio.org/biojava/biojava-live/trunk)
Dasobert: checked out head of trunk (http://www.derkholm.net/svn/repos/dasobert/trunk)
DasRegistry: checked out head of openid branch, as per disccusion with Jonathon 
              (http://www.derkholm.net/svn/repos/dasregistry/branches/openid)
    To build dasregistry, started new NetBeans project as "Web Application with Existing Sources"
                rather than free-form project, because having problems with checked in build.xml file
                this way, NetBeans creates a new build file (nbbuild.xml)

JARs required but not present in repository:
mysql-connector-java-5.1.7-bin.jar (MySQL Connector/J JDBC driver)
axis.jar (Axis 1.4)
jaxrpc.jar
biojava.jar (head of biojava trunk)
dasobert.jar (head of dasobert trunk)
cewolf-1.0.jar
jfreechart-1.0.10.jar
commons-logging-1.0.4.jar (Apache commons)
mail.jar (javamail-1.41)
ehcache-1.6.0-beta3.jar
ols-client.jar (Ontology Lookup Service)
java-openid-sxip-0.9.4.jar
commons-codec-1.3.jar (Apache commons)
commons-httpclient-3.0.1.jar (Apache commons)
commons-logging-1.03.jar (Apache commons)
htmlparser.jar (from openid-sxip)
openxri-client.jar (from openid-sxip)
openxri-syntax.jar (from openid-sxip)
log4j-1.2.15.jar (Apache log4j)
dom3-xercesImpl.jar (from openid-sxip endorsed libs)
dom3-xml-apis.jar (from openid-sxip endorsed libs)
xalan-2.6.0.jar (from openid-sxip endorsed libs)
commons-fileupload-1.2.1.jar (Apache commons)
jcommon-1.0.13.jar (from jfreechart)

Code changes required for registry compilation:

Couldn't find ontology lookup jar that corresponded with naming in head of registry oid branch: The only compilation problem I had was with the DasRegistryOntologyLookup class, which depends on classes in the uk.ac.ebi.www.ontology_lookup.OntologyQuery package. Which I couldn't locate. The closest I could find was the Ontology Lookup Service code at http://www.ebi.ac.uk/ontology-lookup/. As near as I can tell this has the exact same classes as uk.ac.ebi.www.ontology_lookup.OntologyQuery, except they're in a different package: uk.ac.ebi.ook.web.services. So I'm guessing there was just a package refactoring at some point and the registry is using either a newer or older version of the ontology-lookup code than what is available at the OLS site. Changed the imports in DasRegistryOntologyLookup to use the package names in the OLS jar and now everything compiles. Should check with Jonathon and see if where to find ontology_lookup.OntologyQuery package, and whether I should swap in.

Changes required to avoid runtime errors:
    das_registry
        org.biojava.services.das.dao.DasSourceManager.getCapabilities() method
        had to add "sources" column to SELECT statement, otherwise get "column 'sources' not found" errors when accessing ResultSet

    Tomcat / webapp setup:
    JNDI stuff in Tomcat/webapp couldn't find mysql JDBC drivers even when they
        were added as project library jars, and in several places under Tomcat $CATALINA_HOME
    had to add a RESOURCE element to context.xml in project Configuration Files:
       <Resource name="jdbc/mysql" auth="Container" type="javax.sql.DataSource"
               maxActive="8" maxIdle="4" maxWait="10000"
               username="javauser" password="javadude" driverClassName="com.mysql.jdbc.Driver"
               url="jdbc:mysql://localhost:3306/dasregistry01?autoReconnect=true"/>
      (see ~/release/install.txt for Andreas' recommendation on a slightly different Resource declaration)

    Servlet package naming in WebContent/WEB-INF/web.xml is out of sync with source code, so fixed in web.xml:
    (all are just changing "servlet" to "servlets" in servlet-class element class names
    org.biojava.services.das.servlet.Das2RegistryServlet --> org.biojava.services.das.servlets.Das2RegistryServlet
    org.biojava.services.das.servlet.CoordSysServlet --> org.biojava.services.das.servlets.CoordSysServlet
    org.biojava.services.das.servlet.FileUploadServlet --> org.biojava.services.das.servlets.FileUploadServlet
    org.biojava.services.das.servlet.DasSourceActionServlet --> org.biojava.services.das.servlets.DasSourceActionServlet
    org.biojava.services.das.servlet.ProjectIconServlet --> org.biojava.services.das.servlets.ProjectIconServlet
    org.biojava.services.das.servlet.ProjectActionServlet --> org.biojava.services.das.servlets.ProjectActionServlet

WARNING!! Xerces parser problems (org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found) conflicts between JDK1.5 internal Xerces (crimson?) and explicit setting of implementations in org.biojava.services.das.registry.RegistryConfiguration (also explicitly set often in dasobert examples, Das1Validator, some .jsp files, and tests)

Problem is that internal Xerces is auto-overriding any external Xerces (in my case from dom3-xercesImpl.jar) in classpath but with different package naming, which gives errors like:

javax.xml.parsers.FactoryConfigurationError: Provider org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found

See http://www.mbaworld.com/docs/class-loader-howto.html , section on XML parser class loading and

http://java.sun.com/j2se/1.5.0/docs/guide/standards/index.html for endorsed libs mechanism for overriding internal JDK/JRE classes

Unfortunately can't just use endorsed libs in a straightforward ways because there is also a problem with NetBeans using anything other than the JDK1.5 internal lib, which it seems to interpret as having Xalan in the classpath, which causes NetBeans to refuse to run at all: http://wiki.netbeans.org/FaqXalanOnCPmplementations

Tried several ways of using endorsed libs mechanism in combination with using different JDK to run NetBeans (to avoid Xalan problem mentioned above), but never could get the combination to stick on NetBeans6.5 under OSX -- possibly the problem is in specifying the java.endorsed.dirs system property?

SOLUTION: for now, commented out explicit setting of factory in RegistryConfiguration

NEED TO EITHER COME UP WITH ALTERNATIVE, OR ALSO FIX IN OTHER CLASSES


Registry MySQL database:

In general, preface everything on command-line with sudo in order to admin stuff... Alternatively, for mysql command line login as root:

       mysql -u root -p

Make sure MySQL is installed

Starting up MySQL daemon (as per below):

   sudo /usr/local/mysql/bin/mysqld_safe asks for password
   [Cnt-Z to stop process]
   bg to restart process in background

Set password for root, admin, to standard internal via command-line:

   sudo mysqladmin -u root password ********

Create das2registry01 database:

   sudo /usr/local/mysql/bin/mysqladmin create dasregistry01
Populated database from registry SQL file that AndyJ sent me
    (which I think is a mysqldump of the current registry database,
         file is at ~/projects/external/das/das_registry_oid_branch/manually_added_db/reg.sql)         

sudo mysql -p dasregistry01 < reg.sql


Added user: "javauser", password: "javadude" via mysql client:

   -> grant all privileges
   ->  on *.* to 'javauser'@'localhost'
   ->  identified by 'javadude'
   ->  with grant option;

first had to set root/admin password, otherwise got error:

  "Access denied for user @'localhost' (using password: NO)"

see discussion at ??? on problem with using MySQL JDBC drivers without password

To see users:

  select host, user, password from mysql.user

Use DbVisualizer application to browse database:

   current connection alias: "local MySQL"
   database URL: jdbc:mysql://localhost:3306/
        (or optionally to see _just_ dasregistry01 database:
               jdbc:mysql://localhost:3306/dasregistry01 )
   userid: root (or optionally "javauser")
   password:  ****** (or optionally "javadude")

Older installation notes in code repository

Only after figuring out all of the above did I find Andreas' notes from Nov 2005 on local installation of the DAS registry, in the code repository at
    $CHECKOUT_ROOT/release/install.txt
Also the SQL for creating an empty database registry:
    $CHECKOUT_ROOT/release/dasregistry.sql
And the (old and out of date) war file for easy distribution
    $CHECKOUT_ROOT/release/dasregistry.war
WARNING!!!  The installation notes, SQL file, and .war are out of date,
    at least for the openid branch --
         for example:
            the SQL does not include creation of "sources" and "interaction" columns in the "registry" tab
            the .war does not include _anything_ from Dasobert
There are additional jars in $CHECKOUT_ROOT/release/version2.0/ that date from July 2006
        biojavar.jar, dasobert.jar, dasregistry.jar
        These are closer to current code base, for example there are Das2Source etc. classes which were not present in Nov2005 war
        But still out of date -- for instance nothing on molecular interactions in dasobert jar
THIS STUFF NEEDS UPDATING!!!
STILL NEED TO DO: the installation notes suggest putting the RESOURCE configuration for jdbc/mysql in the $TOMCAT/conf/server.xml config file instead of in the webapp context.xml file as I did above

Remaining Issues

Still getting warning messages about log4j from Tomcat when registry webapp is deployed: log4j:WARN No appenders could be found for logger (org.springframework.context.support.ClassPathXmlApplicationContext). log4j:WARN Please initialize the log4j system properly.

Problems with das.xsl stylesheet

    Browser is having trouble retrieving the das.xsl stylesheet
    Because of the way the webapp is configured (mapping das1/*) and the way the stylesheet is specified in the sources XML (just as "das.xsl"), the browser does a GET on [rootpath]/das1/das.xsl.  The server, rather than directly returning the das.xsl file for this request, hands off responsibility to the Das2RegistryServlet.  Which retrieves the file as a resource stream from the servlet context and then streams it back as content of the HTTP response.  But resource-not-found exceptions are getting thrown.  Looks like problem is that at least some combinations of JVM/TomCat really want a "/" prefixed to the file name in arg to the ServletContext.getResourceAsStream() call (see http://www.xyzws.com/servletfaq/how-to-use-servletcontextgetresourceasstream/18)

SOLUTION: added conditional so that if "das.xsl" doesn't work for resource retrieval, add a "/" prefix and try again.

Also repository is missing a bunch of css stylesheet files, referred to in the das.xsl (and therefore like das.xsl, they are triggering attempts to load as resources in Das2RegistryServlet calls (in response to browser GETs).

   /css/dazzle.css
   /css/content.css
   /css/content.css
   /css/printer-styles.css
   /css/screen-styles.css

TEMPORARY SOLUTION: turned off adding of xml-stylesheet clause to sources XML response in ServletResponseWriter (made adding it a boolean option)

NEED TO FIX: once css is checked into repository, can flip boolean to turn xml-stylesheet back on)

Looking at adding support for DAS2 capabilities

Starting to modify to support DAS2 "segments", "types", "features" capabilities (and extend "sources" support)

Added columns to registry database for das2:segments, das2:types, das2:features mysql -u root -p

 > use dasregistry01;
 > alter table registry add das2_segments bit;
 > alter table registry add das2_types bit;
 > alter table registry add das2_features bit;

org.biojava.dasobert.das.Capabilities

     added constants/enums for DAS2 segments, types, features capabilities

org.biojava.services.das.dao.DasSourceManager

     changed how get_capabilities() database query is constructed, so it is based on capabilities specified in org.biojava.dasobert.das.Capabilities constants, rather than having to be manually synchronized with Capabilities.  So addition of DAS2 capabilities constants to Capabilities is automatically reflected in DasSourceManager.  This also fixes previous runtime error problem with "sources" absence in database query, but in a more general way than original fix mentioned above.

Major problem going forward is that the registry database doesn't store capability URLs, but rather a URL for the versioned source and a set of BIT/boolean columns for whether a versioned source supports a particular capability. As near as I can tell the query_uri of the capability is then composed from the versioned source URI and the column name corresponding to a capability.

It looks like there was work done by Andreas on a revised registry database schema that supported decoupling of capability URIs from versioned source URIs via source, capability, and source2capability tables. Is there an instance of this database still around?