From morganmacleod at gmail.com  Thu Feb  4 16:52:11 2010
From: morganmacleod at gmail.com (Morgan MacLeod)
Date: Thu, 4 Feb 2010 17:52:11 -0500
Subject: [Users] Compiling CactusElliptic/EllipticPETSc with PETSc 3.0.0
Message-ID: <49db774a1002041452u7ca326c3p95d7d6f04a01f59c@mail.gmail.com>

Hello All,

I'm just getting started working with Cactus and have a newbie question:

I'm curious if anyone has compiled the EllPETSc thorn with an installation
of PETSc 3.0.0? What did doing so require?

In petsc_flat_solver.c and in petsc_confmetric_solver.c there is a single
line which calls the depricated PETSc function MAT_NO_NEW_NONZERO_LOCATIONS.
By commenting this line, the rest of the source code seems to compile, but
at as the executable is being built I get a long list of errors along these
lines:

/usr/local/petsc/lib/
libpetscksp.a(bcgsl.o): In function `KSPSolve_BCGSL':
/mnt/common/src/petsc/petsc-3.0.0-p2/src/ksp/ksp/impls/bcgsl/bcgsl.c:183:
undefined reference to `dpotrf_'
/mnt/common/src/petsc/petsc-3.0.0-p2/src/ksp/ksp/impls/bcgsl/bcgsl.c:189:
undefined reference to `dcopy_'
/mnt/common/src/petsc/petsc-3.0.0-p2/src/ksp/ksp/impls/bcgsl/bcgsl.c:190:
undefined reference to `dpotrs_'

...

I'm curious if this has something to do with the petsc_defines.h file, where
I notice there isn't an explicit set of definitions for PETSc 3.0?

I'd very much appreciate suggestions about the best way to proceed.

Thank you,

Morgan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.cactuscode.org/pipermail/users/attachments/20100204/e94c02c7/attachment.html 

From schnetter at cct.lsu.edu  Thu Feb  4 17:46:10 2010
From: schnetter at cct.lsu.edu (Erik Schnetter)
Date: Thu, 4 Feb 2010 17:46:10 -0600
Subject: [Users] Compiling CactusElliptic/EllipticPETSc with PETSc 3.0.0
In-Reply-To: <49db774a1002041452u7ca326c3p95d7d6f04a01f59c@mail.gmail.com>
References: <49db774a1002041452u7ca326c3p95d7d6f04a01f59c@mail.gmail.com>
Message-ID: <1153EDEA-195C-4227-9467-88053C934A68@cct.lsu.edu>

On Feb 4, 2010, at 16:52 , Morgan MacLeod wrote:

> Hello All,
> 
> I'm just getting started working with Cactus and have a newbie question: 
> 
> I'm curious if anyone has compiled the EllPETSc thorn with an installation of PETSc 3.0.0? What did doing so require?
> 
> In petsc_flat_solver.c and in petsc_confmetric_solver.c there is a single line which calls the depricated PETSc function MAT_NO_NEW_NONZERO_LOCATIONS. By commenting this line, the rest of the source code seems to compile, but at as the executable is being built I get a long list of errors along these lines:
> 
> /usr/local/petsc/lib/
> libpetscksp.a(bcgsl.o): In function `KSPSolve_BCGSL':
> /mnt/common/src/petsc/petsc-3.0.0-p2/src/ksp/ksp/impls/bcgsl/bcgsl.c:183: undefined reference to `dpotrf_'
> /mnt/common/src/petsc/petsc-3.0.0-p2/src/ksp/ksp/impls/bcgsl/bcgsl.c:189: undefined reference to `dcopy_'
> /mnt/common/src/petsc/petsc-3.0.0-p2/src/ksp/ksp/impls/bcgsl/bcgsl.c:190: undefined reference to `dpotrs_'
> 
> ...
> 
> I'm curious if this has something to do with the petsc_defines.h file, where I notice there isn't an explicit set of definitions for PETSc 3.0?  
> 
> I'd very much appreciate suggestions about the best way to proceed. 


Morgan

The development version of Cactus should already contain the necessary changes to EllPETSc to handle PETSc 3.0.  (The Cactus web site should tell you how to obtain this version; you will probably have to exchange both the flesh and all Cactus* thorns.)

The linker error that you see happens because some LAPACK routines are not found.  It could be that you are not building with LAPACK, or that Cactus mis-detects the C/Fortran inter-language calling conventions.  I just see that PETSc does not require the LAPACK library, which it should.

Do you have LAPACK installed on your system?  Can you try including the thorns CactusExternal/LAPACK and CactusExternal/BLAS?  You will also have to set the options specifying where these libraries are installed.

-erik

-- 
Erik Schnetter <schnetter at cct.lsu.edu>   http://www.cct.lsu.edu/~eschnett/





From schnetter at cct.lsu.edu  Sat Feb  6 11:52:30 2010
From: schnetter at cct.lsu.edu (Erik Schnetter)
Date: Sat, 6 Feb 2010 11:52:30 -0600
Subject: [Users] AEIThorns/Formaline updated
Message-ID: <E0F6E691-42FE-416B-9702-F4A66CE48574@cct.lsu.edu>

Thorn AEIThorns/Formaline stores the source code that was used to build an executable.  This can make simulations reproducible, since this makes it more difficult to lose the exact source code.  Source code repositories are in practice not quite the right mechanism for this, since one may lack write permission, or some changes may be only experimental.

Formaline supports two mechanisms for this.  One mechanism creates tarballs of the source code which are included in the binary, and these tarballs can be written out when a simulation is run.

The other mechanism uses git repositories.  This is enabled automatically if git is installed on the build system.  The git repository is created within the source directory, and since git is very efficient in compressing source code and source code changes, it is easily possible to store hundreds or thousands of revisions.  (Personally, I am approaching the ten thousandth build stored in a single git repository.)

With newer versions of git, Formaline used to have a problem with symbolic links pointing to arrangements or thorns.  I have just updated Formaline to work with modern versions of git as well.  I also moved the code that calls git out of the makefile and into perl scripts of their own, so that the code is now easier to follow.

Please give the updated Formaline a try, and report any problems you may encounter.

-erik

-- 
Erik Schnetter <schnetter at cct.lsu.edu>   http://www.cct.lsu.edu/~eschnett/





From morganmacleod at gmail.com  Mon Feb  8 16:42:38 2010
From: morganmacleod at gmail.com (Morgan MacLeod)
Date: Mon, 8 Feb 2010 17:42:38 -0500
Subject: [Users] Boundary conditions with EllPETSc Thorn
Message-ID: <49db774a1002081442y6850c918jdeb48f9b80a7478b@mail.gmail.com>

Hello All,

Thank you for the help getting PETSc to compile with Cactus on my system.

I'm looking to create a simple, isolated, example of solving the Poisson
equation for the potential of a uniform density sphere (similar to the
IDScalarWaveElliptic thorn). My overall goal is to create a progression of
undergrad-level examples surrounding the solution to elliptic equations in
Cactus. I will happily make these available to the list if/when anyone else
is  interested.

I've been successful calling the SOR solver and getting results which match
the analytic solution. When I call the petsc solver, my results are
significantly off. I believe this is because the Boundary conditions are not
being applied. I apply them as:

  ierr = Ell_SetStrKey("yes","EllLinFlat::Bnd::Robin");
  if(ierr<0) CCTK_WARN(1,"Robin boundary key NOT set");

  ierr = Ell_SetRealKey(0.0,"EllLinFlat::Bnd::Robin::inf");
  if(ierr<0) CCTK_WARN(1,"Boundary constant asymptote key NOT set");

  ierr = Ell_SetIntKey (1,"EllLinFlat::Bnd::Robin::falloff");
  if(ierr<0) CCTK_WARN(1,"Robin Boundary Falloff not set");


within my code. Changing, for example, Robin::inf, has an effect on the SOR
solution, but not on the petsc solution.

I finally call the Elliptic solver like this:

ierr = Ell_LinFlatSolver(cctkGH,
               iphi,
               iMcoeff,
               iNcoeff,
               AbsTol,
               RelTol,
               solver);

How should I be setting boundary conditions in my code when I call petsc?

I've attached the source code of my thorn, as well as its ccl files and a
parameter file -- hopefully this will be enough information, but please let
me know if I'm being unclear.

Thank you again for the help.

Morgan MacLeod
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.cactuscode.org/pipermail/users/attachments/20100208/4a47391a/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SolveEll.c
Type: text/x-csrc
Size: 5496 bytes
Desc: not available
Url : http://www.cactuscode.org/pipermail/users/attachments/20100208/4a47391a/attachment-0001.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: interface.ccl
Type: application/octet-stream
Size: 304 bytes
Desc: not available
Url : http://www.cactuscode.org/pipermail/users/attachments/20100208/4a47391a/attachment-0004.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: param.ccl
Type: application/octet-stream
Size: 229 bytes
Desc: not available
Url : http://www.cactuscode.org/pipermail/users/attachments/20100208/4a47391a/attachment-0005.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: schedule.ccl
Type: application/octet-stream
Size: 320 bytes
Desc: not available
Url : http://www.cactuscode.org/pipermail/users/attachments/20100208/4a47391a/attachment-0006.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SolveEllPETSc.par
Type: application/octet-stream
Size: 1424 bytes
Desc: not available
Url : http://www.cactuscode.org/pipermail/users/attachments/20100208/4a47391a/attachment-0007.obj 

From schnetter at cct.lsu.edu  Sun Feb 14 12:34:17 2010
From: schnetter at cct.lsu.edu (Erik Schnetter)
Date: Sun, 14 Feb 2010 12:34:17 -0600
Subject: [Users] Change to symmetry thorns
Message-ID: <5B258F56-9EAA-45A0-8EB4-6BF7BE803918@cct.lsu.edu>

I have just applied a change to the symmetry thorns ReflectionSymmetry, RotatingSymmetry90, and RotatingSymmetry180.  Previously, they would determine the number of symmetry zones from the number of ghost zones.  To add flexibility and to clean up the mechanism, these thorns now query CoordBase for the number of boundary zones on the corresponding faces.

This may break existing parameter files if you are using any of these three symmetry thorns, and if you do not use CoordBase to specify the size of the domain.  In this case, you will need to set the corresponding CoordBase parameters in your parameter files:

# Assuming that you are using 3 ghost zones
# Assuming that you are using octant mode, with 3 symmetry faces at the lower domain boundary faces:
CoordBase::boundary_size_x_lower = 3
CoordBase::boundary_size_y_lower = 3
CoordBase::boundary_size_z_lower = 3

The thorn guide of CactusBase/CoordBase contains more details and further examples.

-erik

-- 
Erik Schnetter <schnetter at cct.lsu.edu>   http://www.cct.lsu.edu/~eschnett/





From fahadpro at gmail.com  Mon Feb 15 08:21:26 2010
From: fahadpro at gmail.com (fahad nasir)
Date: Mon, 15 Feb 2010 19:21:26 +0500
Subject: [Users] Evolution of Schwarzschild Initial Data How?
Message-ID: <db540a6e1002150621t2d48043ai91139965133204e0@mail.gmail.com>

Hi,

Im not sure is this the correct forum to ask this question. Well coming to
the point. I have evolved a simple static Schwarzschild initial data and
evolved wiht ADM formalism. The simple parameter file is as under..


admbase::evolution_method = "ADM"
adm::method= "stagleap"
admbase::initial_lapse = "one"
admbase::lapse_evolution_method = "1+log"

admbase::metric_type = "static conformal"
admbase::initial_data = "schwarzschild"
idanalyticbh::mass = 1.0


Im still confused how can you evolved Schwarzschild initial data (which
surely a static solution), even in isotropic coordinates. I was also able to
extract Weyl scalars which shows signs of some kind of waves. Surely this is
not due to lapse because Weyl scalar is gauge invariant quantity. My is
question is simple..

Is this because of black hole is initially perturbed initially ?

Or there is some kind of numerical stability  which grows in time ?

Or some kind of non-symmetry of black hole ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.cactuscode.org/pipermail/users/attachments/20100215/69dd2467/attachment.html 

From ian.hinder at aei.mpg.de  Mon Feb 15 08:41:27 2010
From: ian.hinder at aei.mpg.de (Ian Hinder)
Date: Mon, 15 Feb 2010 15:41:27 +0100
Subject: [Users] Evolution of Schwarzschild Initial Data How?
In-Reply-To: <db540a6e1002150621t2d48043ai91139965133204e0@mail.gmail.com>
References: <db540a6e1002150621t2d48043ai91139965133204e0@mail.gmail.com>
Message-ID: <B4AEF3EC-6627-4052-8C59-E8543CAFF4E5@aei.mpg.de>


On 15 Feb 2010, at 15:21, fahad nasir wrote:

> Hi,
>
> Im not sure is this the correct forum to ask this question. Well  
> coming to the point. I have evolved a simple static Schwarzschild  
> initial data and evolved wiht ADM formalism. The simple parameter  
> file is as under..
>
>
> admbase::evolution_method = "ADM"
> adm::method= "stagleap"
> admbase::initial_lapse = "one"
> admbase::lapse_evolution_method = "1+log"
>
> admbase::metric_type = "static conformal"
> admbase::initial_data = "schwarzschild"
> idanalyticbh::mass = 1.0
>
>
> Im still confused how can you evolved Schwarzschild initial data  
> (which surely a static solution), even in isotropic coordinates.

It depends on the gauge you use.  Whilst the initial data is in  
isotropic coordinates, if you evolve with 1+log lapse the evolved  
solution will no longer be in Schwarzschild isotropic coordinates, as  
the lapse you compute from the isotropic Schwarzschild metric does not  
satisfy the 1+log lapse condition.

> I was also able to extract Weyl scalars which shows signs of some  
> kind of waves. Surely this is not due to lapse because Weyl scalar  
> is gauge invariant quantity. My is question is simple..
>
> Is this because of black hole is initially perturbed initially ?
>
> Or there is some kind of numerical stability  which grows in time ?
>
> Or some kind of non-symmetry of black hole ?

You could check whether it is a numerical effect by computing the  
solution at different resolutions.  If the Weyl scalars converge to  
zero then you are just seeing finite differencing error.  If they  
appear to "blow-up" as you increase the resolution, then you are  
probably seeing an instability, but I would be surprised at this  
(though this is ADM, which will be unstable in general).

If there are "gauge waves" in your solution, which I would expect from  
a 1+log slicing condition, then while the Weyl scalars should be  
constant at a given areal coordinate R, your solution coordinate r  
will be different to the areal coordinate R.  I'm not sure if this is  
the reason for the waves that you see though.

-- 
Ian Hinder
ian.hinder at aei.mpg.de