hello Newbie questions about galaxy and clustering

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

hello Newbie questions about galaxy and clustering

leconte
hello
   I have installed an galaxy instance for purpose testing on a 8 CPU
server without problem all run perfectly.
   I have made some galaxy xml wrapper to understant how galaxy calls
program and other little things.


   Now I want to install a Galaxy server with a cluster configuration.


   Before I have tested HTCondor but I have no Idea how I can mix Galaxy
and HTCondor.

   The perfect thing was somebody have already tested this type of condor.

   But I can satisfy myself with any other cluster type.

   What I don't understant in fact ( maybe my worst problem ) where lie
all parts of this puzzle.

  I explain.

   I know that I need Galaxy. I know too I need a cluster  ( in my case
HTCondor )
   I understand  I need something between these 2 parts ... I think
something like PBS or SGE but it's not totaly clear for me

I'm search on galaxy mailling list archive ... but I just found 2 mails
in 2008 :(

has anybody made this conf or have a configuration ?

greetings

PS : sorry for my poor english :)
___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:

  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Question about scripting

leconte
hello,

My subject is at the border between Galaxy and Condor

I'found method to use Galaxy with condor cluster solution.

In fact I have wrote a script when Galaxy call clustalw2 , it calls in
fact a script called clustalw2.

All run exactly as I want excepting  ... but Galaxy get in error and
said there was a problem


I turn my script to debug mode and it puts followings output

+ prog=/usr/bin/clustalw2
+ err=0
+ infile=-infile=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat
+ outfile=-outfile=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/galaxy_dataset_80.dat
+ outorder=-OUTORDER=ALIGNED
+ seqnos=-SEQNOS=OFF
+ typeseq=-TYPE=DNA
+ var6=
+ var7=
+ var8=
+ var9=
+ echo

++ echo -infile=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat
++ sed -e s/-infile=//
+ infile2=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat
++ sed -e 's/galaxy_dataset_\(.*\)\.dat//'
++ sed -e s/-outfile=//
++ echo -outfile=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/galaxy_dataset_80.dat
+ WorkingDir=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/
+ mkdir -p /home/galaxy/galaxy-dist/database/job_working_directory/000/46/
+ mkdir -p /home/galaxy/galaxy-dist/database/job_working_directory/000/46/
+ mkdir -p /home/galaxy/galaxy-dist/database/job_working_directory/000/46/
+ [[ -n /usr/bin/clustalw2==clustalw2 ]]
+ echo 'universe                = vanilla'
+ echo 'executable              = /usr/bin/mpirun '
+ echo 'arguments               = "-np 10 -hostfile /home/galaxy/hostfile /usr/bin/clustalw-mpi -infile=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat -outfile=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/galaxy_dataset_80.dat  -OUTORDER=ALIGNED  -SEQNOS=OFF -TYPE=DNA "'
+ echo 'Getenv                  = True
machine_count           = 1
transfer_input_files    = /home/galaxy/hostfile,/usr/bin/clustalw-mpi, /home/galaxy/galaxy-dist/database/files/000/dataset_11.dat
should_transfer_files   = yes
Log                     = /home/galaxy/logs/normal.$(cluster).$(process).log
Output                  = /home/galaxy/logs/normal.$(cluster).$(process).out
Error                   = /home/galaxy/logs/normal.$(cluster).$(process).error
notification            = never
queue 1'
+ /usr/bin/condor_submit /home/galaxy/submit
Submitting job(s).
1 job(s) submitted to cluster 179.




I suspect the method I use, doesn't return properly information to
galaxy ... but I don't know what I can do ...

Any Idea ...
___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:

  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Question about scripting

Nate Coraor
Hello,

I've moved this over to the galaxy-dev list since it concerns a local installation.

You'll want to use the 'condor' job running plugin, which is provided with the Galaxy distribution.  Have a look at the job_conf.xml.sample_advanced file to see how this would be configured.

A very basic job_conf.xml that should submit all jobs to HTCondor:

<?xml version="1.0"?>
<job_conf>
    <plugins workers="4">
        <plugin id="condor" type="runner" load="galaxy.jobs.runners.condor:CondorJobRunner" />
    </plugins>
    <destinations default="condor">
        <destination id="condor" runner="condor"/>
    </destinations>
</job_conf>

--nate

On Apr 19, 2013, at 8:32 AM, leconte wrote:

> hello,
>
> My subject is at the border between Galaxy and Condor
>
> I'found method to use Galaxy with condor cluster solution.
>
> In fact I have wrote a script when Galaxy call clustalw2 , it calls in fact a script called clustalw2.
>
> All run exactly as I want excepting  ... but Galaxy get in error and said there was a problem
>
>
> I turn my script to debug mode and it puts followings output
>
> + prog=/usr/bin/clustalw2
> + err=0
> + infile=-infile=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat
> + outfile=-outfile=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/galaxy_dataset_80.dat
> + outorder=-OUTORDER=ALIGNED
> + seqnos=-SEQNOS=OFF
> + typeseq=-TYPE=DNA
> + var6=
> + var7=
> + var8=
> + var9=
> + echo
>
> ++ echo -infile=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat
> ++ sed -e s/-infile=//
> + infile2=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat
> ++ sed -e 's/galaxy_dataset_\(.*\)\.dat//'
> ++ sed -e s/-outfile=//
> ++ echo -outfile=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/galaxy_dataset_80.dat
> + WorkingDir=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/
> + mkdir -p /home/galaxy/galaxy-dist/database/job_working_directory/000/46/
> + mkdir -p /home/galaxy/galaxy-dist/database/job_working_directory/000/46/
> + mkdir -p /home/galaxy/galaxy-dist/database/job_working_directory/000/46/
> + [[ -n /usr/bin/clustalw2==clustalw2 ]]
> + echo 'universe                = vanilla'
> + echo 'executable              = /usr/bin/mpirun '
> + echo 'arguments               = "-np 10 -hostfile /home/galaxy/hostfile /usr/bin/clustalw-mpi -infile=/home/galaxy/galaxy-dist/database/files/000/dataset_11.dat -outfile=/home/galaxy/galaxy-dist/database/job_working_directory/000/46/galaxy_dataset_80.dat  -OUTORDER=ALIGNED  -SEQNOS=OFF -TYPE=DNA "'
> + echo 'Getenv                  = True
> machine_count           = 1
> transfer_input_files    = /home/galaxy/hostfile,/usr/bin/clustalw-mpi, /home/galaxy/galaxy-dist/database/files/000/dataset_11.dat
> should_transfer_files   = yes
> Log                     = /home/galaxy/logs/normal.$(cluster).$(process).log
> Output                  = /home/galaxy/logs/normal.$(cluster).$(process).out
> Error                   = /home/galaxy/logs/normal.$(cluster).$(process).error
> notification            = never
> queue 1'
> + /usr/bin/condor_submit /home/galaxy/submit
> Submitting job(s).
> 1 job(s) submitted to cluster 179.
>
>
>
>
> I suspect the method I use, doesn't return properly information to galaxy ... but I don't know what I can do ...
>
> Any Idea ...
> ___________________________________________________________
> The Galaxy User list should be used for the discussion of
> Galaxy analysis and other features on the public server
> at usegalaxy.org.  Please keep all replies on the list by
> using "reply all" in your mail client.  For discussion of
> local Galaxy instances and the Galaxy source code, please
> use the Galaxy Development list:
>
> http://lists.bx.psu.edu/listinfo/galaxy-dev
>
> To manage your subscriptions to this and other Galaxy lists,
> please use the interface at:
>
> http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>
> http://galaxyproject.org/search/mailinglists/


___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org.  Please keep all replies on the list by
using "reply all" in your mail client.  For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:

  http://lists.bx.psu.edu/listinfo/galaxy-dev

To manage your subscriptions to this and other Galaxy lists,
please use the interface at:

  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:

  http://galaxyproject.org/search/mailinglists/