From yfeng1 at tigers.lsu.edu Sun Aug 2 22:00:07 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Sun, 2 Aug 2009 22:00:07 -0500 Subject: [Users] A Problem about Ghost Zone Message-ID: <1e8c69dc0908022000q2ae9715dn3701d31828873810@mail.gmail.com> Let me take an example to clarify my problem. We have a global system with 8 gridblocks 0 | 1 2 3 4 5 6 7 8 | 9 where 0 and 9 are boundary blocks. When it is separated and sent to 2 processors, we have following two local system: Pro 1 Pro 2 0 | 1 2 3 4 | 5 0 | 1 2 3 4 | 5 (0 and 5 are boundary blocks) 0 | 1 2 3 4 | 5 4 | 5 6 7 8 | 9 Global So, for processor 1, boundary 5 should be the block 1 in processor 2 and for processor 2, boundary 0 should be the block 4 in processor 1. However, in my code, 5 in processor 1 can not share value from 1 in processor 2. It takes initial value. If there is no initial value, it just takes 0. Is there any possible reason for this problem? Thank you in advance! Sincerely, YIN -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.cactuscode.org/pipermail/users/attachments/20090802/10c32db3/attachment.html From knarf at cct.lsu.edu Mon Aug 3 08:41:46 2009 From: knarf at cct.lsu.edu (Frank Loeffler) Date: Mon, 3 Aug 2009 08:41:46 -0500 Subject: [Users] A Problem about Ghost Zone In-Reply-To: <1e8c69dc0908022000q2ae9715dn3701d31828873810@mail.gmail.com> References: <1e8c69dc0908022000q2ae9715dn3701d31828873810@mail.gmail.com> Message-ID: <20090803134145.GL29820@numrel07.cct.lsu.edu> Hi Yin Feng, On Sun, Aug 02, 2009 at 10:00:07PM -0500, Yin Feng wrote: > We have a global system with 8 gridblocks > 0 | 1 2 3 4 5 6 7 8 | 9 where 0 and 9 are boundary blocks. > When it is separated and sent to 2 processors, we have following two local > system: > > Pro 1 Pro 2 > 0 | 1 2 3 4 | 5 0 | 1 2 3 4 | 5 (0 and 5 are boundary blocks) > > 0 | 1 2 3 4 | 5 4 | 5 6 7 8 | 9 Global > > So, for processor 1, boundary 5 should be the block 1 in processor 2 and > for processor 2, boundary 0 should be the block 4 in processor 1. That is correct. > However, in my code, 5 in processor 1 can not share value from 1 in > processor 2. I don't understand what you mean with 'can not share'. The way the example above is used is that at some point during the evolution the grid function is synced, which means the value of block 1 on processor 2 is copied to block 5 of processor 1 and the value of block 4 on processor 1 is copied to block 0 on processor 2 (here I use the term 'block' as you do, we would usually call them grid points). > It takes initial value. If there is no initial value, it just takes 0. This is what is usually done. For initial data you can either initialize all points (blocks) or you can leave out the ghost zone and sync the variable after the initialization. Both will give the same result, but sometimes one or the other method is a bit faster. > Is there any possible reason for this problem? I am sorry, but I do not see the problem. What you described is how the processor decomposition of grid functions works. Why is this a problem for you? Frank Loeffler From bernard.j.kelly at nasa.gov Mon Aug 3 08:58:14 2009 From: bernard.j.kelly at nasa.gov (Bernard Kelly) Date: Mon, 03 Aug 2009 09:58:14 -0400 Subject: [Users] A Problem about Ghost Zone In-Reply-To: <20090803134145.GL29820@numrel07.cct.lsu.edu> References: <1e8c69dc0908022000q2ae9715dn3701d31828873810@mail.gmail.com> <20090803134145.GL29820@numrel07.cct.lsu.edu> Message-ID: <4A76ECF6.7090008@nasa.gov> Yin, are you calling one of the CCTK_Sync routines to synchronise data across processors? Bernard Frank Loeffler wrote: > Hi Yin Feng, > > On Sun, Aug 02, 2009 at 10:00:07PM -0500, Yin Feng wrote: >> We have a global system with 8 gridblocks >> 0 | 1 2 3 4 5 6 7 8 | 9 where 0 and 9 are boundary blocks. >> When it is separated and sent to 2 processors, we have following two local >> system: >> >> Pro 1 Pro 2 >> 0 | 1 2 3 4 | 5 0 | 1 2 3 4 | 5 (0 and 5 are boundary blocks) >> >> 0 | 1 2 3 4 | 5 4 | 5 6 7 8 | 9 Global >> >> So, for processor 1, boundary 5 should be the block 1 in processor 2 and >> for processor 2, boundary 0 should be the block 4 in processor 1. > > That is correct. > >> However, in my code, 5 in processor 1 can not share value from 1 in >> processor 2. > > I don't understand what you mean with 'can not share'. The way the > example above is used is that at some point during the evolution the > grid function is synced, which means the value of block 1 on processor 2 > is copied to block 5 of processor 1 and the value of block 4 on > processor 1 is copied to block 0 on processor 2 (here I use the term > 'block' as you do, we would usually call them grid points). > >> It takes initial value. If there is no initial value, it just takes 0. > > This is what is usually done. For initial data you can either initialize > all points (blocks) or you can leave out the ghost zone and sync the > variable after the initialization. Both will give the same result, but > sometimes one or the other method is a bit faster. > >> Is there any possible reason for this problem? > > I am sorry, but I do not see the problem. What you described is how the > processor decomposition of grid functions works. Why is this a problem > for you? > > Frank Loeffler > > _______________________________________________ > Users mailing list > Users at cactuscode.org > http://www.cactuscode.org/mailman/listinfo/users From yfeng1 at tigers.lsu.edu Mon Aug 3 09:11:31 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Mon, 3 Aug 2009 09:11:31 -0500 Subject: [Users] A Problem about Ghost Zone In-Reply-To: <20090803134145.GL29820@numrel07.cct.lsu.edu> References: <1e8c69dc0908022000q2ae9715dn3701d31828873810@mail.gmail.com> <20090803134145.GL29820@numrel07.cct.lsu.edu> Message-ID: <1e8c69dc0908030711r7efcc1fdp85ec0a70159921b1@mail.gmail.com> Thank you Frank I tried to output some parameters in 5 in processor 1 at each evolution. But, what I found is that all the parameters keep initial value at any time. They are not renew by 1 in processor 2. By the way, is there any way I can control at with point in evolution the grid function is synced and monitor whether this is really done? Thank you! On Mon, Aug 3, 2009 at 8:41 AM, Frank Loeffler wrote: > Hi Yin Feng, > > On Sun, Aug 02, 2009 at 10:00:07PM -0500, Yin Feng wrote: > > We have a global system with 8 gridblocks > > 0 | 1 2 3 4 5 6 7 8 | 9 where 0 and 9 are boundary blocks. > > When it is separated and sent to 2 processors, we have following two > local > > system: > > > > Pro 1 Pro 2 > > 0 | 1 2 3 4 | 5 0 | 1 2 3 4 | 5 (0 and 5 are boundary blocks) > > > > 0 | 1 2 3 4 | 5 4 | 5 6 7 8 | 9 Global > > > > So, for processor 1, boundary 5 should be the block 1 in processor 2 and > > for processor 2, boundary 0 should be the block 4 in processor 1. > > That is correct. > > > However, in my code, 5 in processor 1 can not share value from 1 in > > processor 2. > > I don't understand what you mean with 'can not share'. The way the > example above is used is that at some point during the evolution the > grid function is synced, which means the value of block 1 on processor 2 > is copied to block 5 of processor 1 and the value of block 4 on > processor 1 is copied to block 0 on processor 2 (here I use the term > 'block' as you do, we would usually call them grid points). > > > It takes initial value. If there is no initial value, it just takes 0. > > This is what is usually done. For initial data you can either initialize > all points (blocks) or you can leave out the ghost zone and sync the > variable after the initialization. Both will give the same result, but > sometimes one or the other method is a bit faster. > > > Is there any possible reason for this problem? > > I am sorry, but I do not see the problem. What you described is how the > processor decomposition of grid functions works. Why is this a problem > for you? > > Frank Loeffler > > _______________________________________________ > Users mailing list > Users at cactuscode.org > http://www.cactuscode.org/mailman/listinfo/users > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.cactuscode.org/pipermail/users/attachments/20090803/0fbe4007/attachment.html From knarf at cct.lsu.edu Mon Aug 3 09:18:22 2009 From: knarf at cct.lsu.edu (Frank Loeffler) Date: Mon, 3 Aug 2009 09:18:22 -0500 Subject: [Users] A Problem about Ghost Zone In-Reply-To: <1e8c69dc0908030711r7efcc1fdp85ec0a70159921b1@mail.gmail.com> References: <1e8c69dc0908022000q2ae9715dn3701d31828873810@mail.gmail.com> <20090803134145.GL29820@numrel07.cct.lsu.edu> <1e8c69dc0908030711r7efcc1fdp85ec0a70159921b1@mail.gmail.com> Message-ID: <20090803141822.GM29820@numrel07.cct.lsu.edu> Hi, On Mon, Aug 03, 2009 at 09:11:31AM -0500, Yin Feng wrote: > I tried to output some parameters in 5 in processor 1 at each evolution. > But, what I found is that all the parameters keep initial value at any time. > They are not renew by 1 in processor 2. > > By the way, is there any way I can control at with point in evolution the > grid function is synced and monitor whether this is really done? You have to specify when variables are synced, because that is usually an expensive operation (because it involves network traffic). You can do that in schedule.ccl in the block which calls a function. The specified variables will then be synced after the function returns. Take a look at this example: schedule function_name AT CCTK_EVOL { LANG: C SYNC: the_variable_which_should_be_synced SYNC: another_variable_which_should_be_synced SYNC: a_group_of_variables_which_should_be_synced } For this example, the specified variables will be synced after the function "function_name" returned. Frank From yfeng1 at tigers.lsu.edu Tue Aug 4 13:31:40 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Tue, 4 Aug 2009 13:31:40 -0500 Subject: [Users] How to define global variables across all the processors in Cactus? Message-ID: <1e8c69dc0908041131vfd4a778mbeaa7a655e254501@mail.gmail.com> Detailed problem is: During parallel computation, I have variable A defined on each processor. Since I want the sum of these As on every processors, I plan to define a global variable using which to collect value of A from each processor. So, how can I define this kind of variable or there is any substitute? Thank you in advance! Sincerely, YIN -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.cactuscode.org/pipermail/users/attachments/20090804/3be0887c/attachment.html From dprideout at gmail.com Tue Aug 4 13:37:32 2009 From: dprideout at gmail.com (David Rideout) Date: Tue, 4 Aug 2009 14:37:32 -0400 Subject: [Users] How to define global variables across all the processors in Cactus? In-Reply-To: <1e8c69dc0908041131vfd4a778mbeaa7a655e254501@mail.gmail.com> References: <1e8c69dc0908041131vfd4a778mbeaa7a655e254501@mail.gmail.com> Message-ID: <1ce81abb0908041137q59c6b845kaa5aa3e06c95bc73@mail.gmail.com> I think I asked Thomas Radke about this not too long ago. I think he said that it is not possible to reduce grid scalars, but you can generate a grid array A with nprocs entries. Each cpu puts its value in A[myproc], and can get the sum with a reduction call. Is this what you want? Cheers, David On Tue, Aug 4, 2009 at 2:31 PM, Yin Feng wrote: > Detailed problem is: > During parallel computation, I have variable A defined on each processor. > Since I want the sum of these As on every processors, I plan to define a > global variable > using which to collect value of A from each processor. > So, how can I define this kind of variable or there is any substitute? > Thank you in advance! > Sincerely, > YIN > _______________________________________________ > Users mailing list > Users at cactuscode.org > http://www.cactuscode.org/mailman/listinfo/users > > From dprideout at gmail.com Tue Aug 4 14:12:29 2009 From: dprideout at gmail.com (David Rideout) Date: Tue, 4 Aug 2009 15:12:29 -0400 Subject: [Users] How to define global variables across all the processors in Cactus? In-Reply-To: <1ce81abb0908041137q59c6b845kaa5aa3e06c95bc73@mail.gmail.com> References: <1e8c69dc0908041131vfd4a778mbeaa7a655e254501@mail.gmail.com> <1ce81abb0908041137q59c6b845kaa5aa3e06c95bc73@mail.gmail.com> Message-ID: <1ce81abb0908041212u41ebab7j678c4e5d79ced4af@mail.gmail.com> Slight clarification: A is an ordinary 'C array' with nproc elements. I do not know how you could declare this in an interface.ccl. So something like this: int nproc, myproc; CCTK_REAL *A, sum; nproc = CCTK_NProcs{cctkGH); myproc = CCTK_MyProc(cctkGH); A = malloc(nprocs*sizeof(CCTK_REAL)); A[myproc] = my_proc_local_value; CCTK_Barrier(); /* Get sum */ CCTK_Reduce(..., A, ..., &sum, ...); (I do not know the details of the reduction api offhand.) Cheers, David On Tue, Aug 4, 2009 at 2:37 PM, David Rideout wrote: > I think I asked Thomas Radke about this not too long ago. > > I think he said that it is not possible to reduce grid scalars, but > you can generate a grid array A with nprocs entries. Each cpu puts > its value in A[myproc], and can get the sum with a reduction call. > > Is this what you want? > > Cheers, > David > > On Tue, Aug 4, 2009 at 2:31 PM, Yin Feng wrote: >> Detailed problem is: >> During parallel computation, I have variable A defined on each processor. >> Since I want the sum of these As on every processors, I plan to define a >> global variable >> using which to collect value of A from each processor. >> So, how can I define this kind of variable or there is any substitute? >> Thank you in advance! >> Sincerely, >> YIN >> _______________________________________________ >> Users mailing list >> Users at cactuscode.org >> http://www.cactuscode.org/mailman/listinfo/users >> >> > From yfeng1 at tigers.lsu.edu Tue Aug 4 15:04:01 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Tue, 4 Aug 2009 15:04:01 -0500 Subject: [Users] How to define global variables across all the processors in Cactus? In-Reply-To: <1ce81abb0908041212u41ebab7j678c4e5d79ced4af@mail.gmail.com> References: <1e8c69dc0908041131vfd4a778mbeaa7a655e254501@mail.gmail.com> <1ce81abb0908041137q59c6b845kaa5aa3e06c95bc73@mail.gmail.com> <1ce81abb0908041212u41ebab7j678c4e5d79ced4af@mail.gmail.com> Message-ID: <1e8c69dc0908041304o4c3d091egfe71bc6d901a566a@mail.gmail.com> Yes, that really makes sense.Thank you, David! On Tue, Aug 4, 2009 at 2:12 PM, David Rideout wrote: > Slight clarification: A is an ordinary 'C array' with nproc elements. > I do not know how you could declare this in an interface.ccl. So > something like this: > > int nproc, myproc; > CCTK_REAL *A, sum; > > nproc = CCTK_NProcs{cctkGH); > myproc = CCTK_MyProc(cctkGH); > > A = malloc(nprocs*sizeof(CCTK_REAL)); > > A[myproc] = my_proc_local_value; > > CCTK_Barrier(); > > /* Get sum */ > CCTK_Reduce(..., A, ..., &sum, ...); > > (I do not know the details of the reduction api offhand.) > > Cheers, > David > > > > On Tue, Aug 4, 2009 at 2:37 PM, David Rideout wrote: > > I think I asked Thomas Radke about this not too long ago. > > > > I think he said that it is not possible to reduce grid scalars, but > > you can generate a grid array A with nprocs entries. Each cpu puts > > its value in A[myproc], and can get the sum with a reduction call. > > > > Is this what you want? > > > > Cheers, > > David > > > > On Tue, Aug 4, 2009 at 2:31 PM, Yin Feng wrote: > >> Detailed problem is: > >> During parallel computation, I have variable A defined on each > processor. > >> Since I want the sum of these As on every processors, I plan to define a > >> global variable > >> using which to collect value of A from each processor. > >> So, how can I define this kind of variable or there is any substitute? > >> Thank you in advance! > >> Sincerely, > >> YIN > >> _______________________________________________ > >> Users mailing list > >> Users at cactuscode.org > >> http://www.cactuscode.org/mailman/listinfo/users > >> > >> > > > _______________________________________________ > Users mailing list > Users at cactuscode.org > http://www.cactuscode.org/mailman/listinfo/users > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.cactuscode.org/pipermail/users/attachments/20090804/dc9d1e6c/attachment.html From schnetter at cct.lsu.edu Wed Aug 5 02:23:06 2009 From: schnetter at cct.lsu.edu (Erik Schnetter) Date: Wed, 5 Aug 2009 08:23:06 +0100 Subject: [Users] How to define global variables across all the processors in Cactus? In-Reply-To: <1e8c69dc0908041131vfd4a778mbeaa7a655e254501@mail.gmail.com> References: <1e8c69dc0908041131vfd4a778mbeaa7a655e254501@mail.gmail.com> Message-ID: <42F65835-E496-46ED-B8FD-ABC8F94109DA@cct.lsu.edu> On Aug 4, 2009, at 19:31:40, Yin Feng wrote: > Detailed problem is: > > During parallel computation, I have variable A defined on each > processor. > Since I want the sum of these As on every processors, I plan to > define a global variable > using which to collect value of A from each processor. > > So, how can I define this kind of variable or there is any substitute? Yin The Cactus (and the MPI) reduction APIs offer this functionality. You can do the following: const CCTK_REAL local_value = A; printf ("local value = %g\n", (double)local_value); const int sum_op = CCTK_ReductionHandle ("sum"); if (sum_op < 0) CCTK_WARN (CCTK_WARN_ABORT, "error"); CCTK_REAL global_value; /* -1 means: make the sum available to all processors */ const int ierr = CCTK_ReduceLocScalar (cctkGH, -1, sum_op, &local_value, &global_value, CCTK_VARIABLE_REAL); if (ierr) CCTK_WARN (CCTK_WARN_ABORT, "error"); printf ("global value = %g\n", (double)global_value); If A is an array instead of a scalar, then there are corresponding other reduction operators for this as well. In this case you have two choices -- the result can either be an array, where each element is the sum of the corresponding elements of the other processors, or the result can be a single scalar that is the sum of all array elements of all processors. -erik -- Erik Schnetter http://www.cct.lsu.edu/~eschnett/ My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from . -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 203 bytes Desc: This is a digitally signed message part Url : http://www.cactuscode.org/pipermail/users/attachments/20090805/444ed91f/attachment-0001.bin From dprideout at gmail.com Wed Aug 5 08:30:27 2009 From: dprideout at gmail.com (David Rideout) Date: Wed, 5 Aug 2009 09:30:27 -0400 Subject: [Users] How to define global variables across all the processors in Cactus? In-Reply-To: <42F65835-E496-46ED-B8FD-ABC8F94109DA@cct.lsu.edu> References: <1e8c69dc0908041131vfd4a778mbeaa7a655e254501@mail.gmail.com> <42F65835-E496-46ED-B8FD-ABC8F94109DA@cct.lsu.edu> Message-ID: <1ce81abb0908050630ne18fb97g17ad1415cc724c45@mail.gmail.com> So for example one can use this to reduce a grid scalar, then. Thanks for the tip. -D On 8/5/09, Erik Schnetter wrote: > On Aug 4, 2009, at 19:31:40, Yin Feng wrote: > >> Detailed problem is: >> >> During parallel computation, I have variable A defined on each >> processor. >> Since I want the sum of these As on every processors, I plan to >> define a global variable >> using which to collect value of A from each processor. >> >> So, how can I define this kind of variable or there is any substitute? > > > Yin > > The Cactus (and the MPI) reduction APIs offer this functionality. You > can do the following: > > const CCTK_REAL local_value = A; > printf ("local value = %g\n", (double)local_value); > > const int sum_op = CCTK_ReductionHandle ("sum"); > if (sum_op < 0) CCTK_WARN (CCTK_WARN_ABORT, "error"); > > CCTK_REAL global_value; > /* -1 means: make the sum available to all processors */ > const int ierr = CCTK_ReduceLocScalar (cctkGH, -1, sum_op, > &local_value, &global_value, CCTK_VARIABLE_REAL); > if (ierr) CCTK_WARN (CCTK_WARN_ABORT, "error"); > > printf ("global value = %g\n", (double)global_value); > > > If A is an array instead of a scalar, then there are corresponding > other reduction operators for this as well. In this case you have two > choices -- the result can either be an array, where each element is > the sum of the corresponding elements of the other processors, or the > result can be a single scalar that is the sum of all array elements of > all processors. > > -erik > > -- > Erik Schnetter http://www.cct.lsu.edu/~eschnett/ > > My email is as private as my paper mail. I therefore support encrypting > and signing email messages. Get my PGP key from . > > > > From yfeng1 at tigers.lsu.edu Wed Aug 5 08:40:35 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Wed, 5 Aug 2009 08:40:35 -0500 Subject: [Users] initialize parameters for HDF5 dataset Message-ID: <1e8c69dc0908050640q1118c899jf6de6aba0c47a7e1@mail.gmail.com> So for as I know, cactus can recovery from HDF5 datafiles from checkpoint.So, how about I do not have checkpoint and just want to initialize my parameters from an existed HDF5 data file? Is that possible to do that by only changing .par file? Thank you in advance! Sincerely, YIN -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.cactuscode.org/pipermail/users/attachments/20090805/e131ff30/attachment.html From schnetter at cct.lsu.edu Wed Aug 5 09:56:32 2009 From: schnetter at cct.lsu.edu (Erik Schnetter) Date: Wed, 5 Aug 2009 15:56:32 +0100 Subject: [Users] How to define global variables across all the processors in Cactus? In-Reply-To: <1ce81abb0908050630ne18fb97g17ad1415cc724c45@mail.gmail.com> References: <1e8c69dc0908041131vfd4a778mbeaa7a655e254501@mail.gmail.com> <42F65835-E496-46ED-B8FD-ABC8F94109DA@cct.lsu.edu> <1ce81abb0908050630ne18fb97g17ad1415cc724c45@mail.gmail.com> Message-ID: Yes. Cactus also offers an equivalent API to reduce grid variables (grid functions and grid scalars), which accept a variable index instead of a pointer and an explicit type specification. -erik On Aug 5, 2009, at 14:30:27, David Rideout wrote: > So for example one can use this to reduce a grid scalar, then. > > Thanks for the tip. > > -D > > > On 8/5/09, Erik Schnetter wrote: >> On Aug 4, 2009, at 19:31:40, Yin Feng wrote: >> >>> Detailed problem is: >>> >>> During parallel computation, I have variable A defined on each >>> processor. >>> Since I want the sum of these As on every processors, I plan to >>> define a global variable >>> using which to collect value of A from each processor. >>> >>> So, how can I define this kind of variable or there is any >>> substitute? >> >> >> Yin >> >> The Cactus (and the MPI) reduction APIs offer this functionality. >> You >> can do the following: >> >> const CCTK_REAL local_value = A; >> printf ("local value = %g\n", (double)local_value); >> >> const int sum_op = CCTK_ReductionHandle ("sum"); >> if (sum_op < 0) CCTK_WARN (CCTK_WARN_ABORT, "error"); >> >> CCTK_REAL global_value; >> /* -1 means: make the sum available to all processors */ >> const int ierr = CCTK_ReduceLocScalar (cctkGH, -1, sum_op, >> &local_value, &global_value, CCTK_VARIABLE_REAL); >> if (ierr) CCTK_WARN (CCTK_WARN_ABORT, "error"); >> >> printf ("global value = %g\n", (double)global_value); >> >> >> If A is an array instead of a scalar, then there are corresponding >> other reduction operators for this as well. In this case you have >> two >> choices -- the result can either be an array, where each element is >> the sum of the corresponding elements of the other processors, or the >> result can be a single scalar that is the sum of all array elements >> of >> all processors. >> >> -erik >> >> -- >> Erik Schnetter http://www.cct.lsu.edu/~eschnett/ >> >> My email is as private as my paper mail. I therefore support >> encrypting >> and signing email messages. Get my PGP key from > pgp.mit.edu>. >> >> >> >> > _______________________________________________ > Users mailing list > Users at cactuscode.org > http://www.cactuscode.org/mailman/listinfo/users -- Erik Schnetter http://www.cct.lsu.edu/~eschnett/ My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from . -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 203 bytes Desc: This is a digitally signed message part Url : http://www.cactuscode.org/pipermail/users/attachments/20090805/cf571a8d/attachment.bin From schnetter at cct.lsu.edu Wed Aug 5 10:01:11 2009 From: schnetter at cct.lsu.edu (Erik Schnetter) Date: Wed, 5 Aug 2009 16:01:11 +0100 Subject: [Users] initialize parameters for HDF5 dataset In-Reply-To: <1e8c69dc0908050640q1118c899jf6de6aba0c47a7e1@mail.gmail.com> References: <1e8c69dc0908050640q1118c899jf6de6aba0c47a7e1@mail.gmail.com> Message-ID: <54E0F6D1-F6EC-45BF-8EFE-C3C53C58DF2B@cct.lsu.edu> On Aug 5, 2009, at 14:40:35, Yin Feng wrote: > So for as I know, cactus can recovery from HDF5 datafiles from > checkpoint. > So, how about I do not have checkpoint and just want to initialize > my parameters from an existed HDF5 data file? > Is that possible to do that by only changing .par file? Yin Yes, this is possible. Cactus as a file reader functionality that you can use for this. It is implemented by the thorn CactusBase/IOUtil, and should be documented there. Basically you list the file names from which you want to read, and list the Cactus variables that you want to read, both in the parameter file. Of course, variables that are not read from file have to be initialised somewhere in the code. -erik -- Erik Schnetter http://www.cct.lsu.edu/~eschnett/ My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from . -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 203 bytes Desc: This is a digitally signed message part Url : http://www.cactuscode.org/pipermail/users/attachments/20090805/dc8b8b76/attachment.bin From yfeng1 at tigers.lsu.edu Wed Aug 5 11:02:48 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Wed, 5 Aug 2009 11:02:48 -0500 Subject: [Users] initialize parameters for HDF5 dataset In-Reply-To: <54E0F6D1-F6EC-45BF-8EFE-C3C53C58DF2B@cct.lsu.edu> References: <1e8c69dc0908050640q1118c899jf6de6aba0c47a7e1@mail.gmail.com> <54E0F6D1-F6EC-45BF-8EFE-C3C53C58DF2B@cct.lsu.edu> Message-ID: <1e8c69dc0908050902o4fd867cbi9d0589343a31fd4d@mail.gmail.com> Thank you, erik!Does that also work under parallel condition? YIN On Wed, Aug 5, 2009 at 10:01 AM, Erik Schnetter wrote: > On Aug 5, 2009, at 14:40:35, Yin Feng wrote: > > So for as I know, cactus can recovery from HDF5 datafiles from checkpoint. >> So, how about I do not have checkpoint and just want to initialize my >> parameters from an existed HDF5 data file? >> Is that possible to do that by only changing .par file? >> > > Yin > > Yes, this is possible. Cactus as a file reader functionality that you can > use for this. It is implemented by the thorn CactusBase/IOUtil, and should > be documented there. > > Basically you list the file names from which you want to read, and list the > Cactus variables that you want to read, both in the parameter file. Of > course, variables that are not read from file have to be initialised > somewhere in the code. > > -erik > > -- > Erik Schnetter http://www.cct.lsu.edu/~eschnett/ > > My email is as private as my paper mail. I therefore support encrypting > and signing email messages. Get my PGP key from . > > > > > _______________________________________________ > Users mailing list > Users at cactuscode.org > http://www.cactuscode.org/mailman/listinfo/users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.cactuscode.org/pipermail/users/attachments/20090805/764672ef/attachment.html From schnetter at cct.lsu.edu Wed Aug 5 11:29:56 2009 From: schnetter at cct.lsu.edu (Erik Schnetter) Date: Wed, 5 Aug 2009 17:29:56 +0100 Subject: [Users] initialize parameters for HDF5 dataset In-Reply-To: <1e8c69dc0908050902o4fd867cbi9d0589343a31fd4d@mail.gmail.com> References: <1e8c69dc0908050640q1118c899jf6de6aba0c47a7e1@mail.gmail.com> <54E0F6D1-F6EC-45BF-8EFE-C3C53C58DF2B@cct.lsu.edu> <1e8c69dc0908050902o4fd867cbi9d0589343a31fd4d@mail.gmail.com> Message-ID: <937F13EE-5A53-408C-BA9A-0CD8DD40CF0B@cct.lsu.edu> Yin Yes; this also works in parallel. -erik On Aug 5, 2009, at 17:02:48, Yin Feng wrote: > Thank you, erik! > Does that also work under parallel condition? > > YIN > > On Wed, Aug 5, 2009 at 10:01 AM, Erik Schnetter > wrote: > On Aug 5, 2009, at 14:40:35, Yin Feng wrote: > > So for as I know, cactus can recovery from HDF5 datafiles from > checkpoint. > So, how about I do not have checkpoint and just want to initialize > my parameters from an existed HDF5 data file? > Is that possible to do that by only changing .par file? > > Yin > > Yes, this is possible. Cactus as a file reader functionality that > you can use for this. It is implemented by the thorn CactusBase/ > IOUtil, and should be documented there. > > Basically you list the file names from which you want to read, and > list the Cactus variables that you want to read, both in the > parameter file. Of course, variables that are not read from file > have to be initialised somewhere in the code. > > -erik > > -- > Erik Schnetter http://www.cct.lsu.edu/~eschnett/ > > My email is as private as my paper mail. I therefore support > encrypting > and signing email messages. Get my PGP key from . > > > > > _______________________________________________ > Users mailing list > Users at cactuscode.org > http://www.cactuscode.org/mailman/listinfo/users > > > _______________________________________________ > Users mailing list > Users at cactuscode.org > http://www.cactuscode.org/mailman/listinfo/users -- Erik Schnetter http://www.cct.lsu.edu/~eschnett/ My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from . -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 203 bytes Desc: This is a digitally signed message part Url : http://www.cactuscode.org/pipermail/users/attachments/20090805/07ff4655/attachment.bin From yfeng1 at tigers.lsu.edu Thu Aug 6 10:00:57 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Thu, 6 Aug 2009 10:00:57 -0500 Subject: [Users] initialize parameters for HDF5 dataset In-Reply-To: <937F13EE-5A53-408C-BA9A-0CD8DD40CF0B@cct.lsu.edu> References: <1e8c69dc0908050640q1118c899jf6de6aba0c47a7e1@mail.gmail.com> <54E0F6D1-F6EC-45BF-8EFE-C3C53C58DF2B@cct.lsu.edu> <1e8c69dc0908050902o4fd867cbi9d0589343a31fd4d@mail.gmail.com> <937F13EE-5A53-408C-BA9A-0CD8DD40CF0B@cct.lsu.edu> Message-ID: <1e8c69dc0908060800p1d0750cav7b5aaf4b102c8725@mail.gmail.com> I followed instructions to add somethings in my .par file: IO::filereader_ID_dir = "/work/yfeng1/BlackOilSim/SPE11/datafiles/" IO::filereader_ID_files = "Pw.h5" IO::filereader_ID_vars = "BlackOilBase::Pressure" where, Pw.h5 is the HDF5 file only contain parameter Pw Pressure is the group name which includes Pw, Po, and Pg when I try to make it run, it shows INFO (IOUtil): Reading variables from data file 'Pw.h5' WARNING[L1,P2] (IOUtil): Failed to read variables from data file 'Pw.h5' So, is there any thing missing in my .par file? Thank you! Sincerely, YIN On Wed, Aug 5, 2009 at 11:29 AM, Erik Schnetter wrote: > Yin > > Yes; this also works in parallel. > > -erik > > > On Aug 5, 2009, at 17:02:48, Yin Feng wrote: > > Thank you, erik! >> Does that also work under parallel condition? >> >> YIN >> >> On Wed, Aug 5, 2009 at 10:01 AM, Erik Schnetter >> wrote: >> On Aug 5, 2009, at 14:40:35, Yin Feng wrote: >> >> So for as I know, cactus can recovery from HDF5 datafiles from checkpoint. >> So, how about I do not have checkpoint and just want to initialize my >> parameters from an existed HDF5 data file? >> Is that possible to do that by only changing .par file? >> >> Yin >> >> Yes, this is possible. Cactus as a file reader functionality that you can >> use for this. It is implemented by the thorn CactusBase/IOUtil, and should >> be documented there. >> >> Basically you list the file names from which you want to read, and list >> the Cactus variables that you want to read, both in the parameter file. Of >> course, variables that are not read from file have to be initialised >> somewhere in the code. >> >> -erik >> >> -- >> Erik Schnetter >> http://www.cct.lsu.edu/~eschnett/ >> >> My email is as private as my paper mail. I therefore support encrypting >> and signing email messages. Get my PGP key from . >> >> >> >> >> _______________________________________________ >> Users mailing list >> Users at cactuscode.org >> http://www.cactuscode.org/mailman/listinfo/users >> >> >> _______________________________________________ >> Users mailing list >> Users at cactuscode.org >> http://www.cactuscode.org/mailman/listinfo/users >> > > > -- > Erik Schnetter http://www.cct.lsu.edu/~eschnett/ > > My email is as private as my paper mail. I therefore support encrypting > and signing email messages. Get my PGP key from . > > > > > _______________________________________________ > Users mailing list > Users at cactuscode.org > http://www.cactuscode.org/mailman/listinfo/users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.cactuscode.org/pipermail/users/attachments/20090806/bf7e0d49/attachment-0001.html From takami at theo.phys.sci.hiroshima-u.ac.jp Thu Aug 6 11:23:04 2009 From: takami at theo.phys.sci.hiroshima-u.ac.jp (KENTARO TAKAMI) Date: Fri, 7 Aug 2009 01:23:04 +0900 Subject: [Users] How to control the number of threads? Message-ID: <4c323ec00908060923n78d591ecy9d00904e8b1fd965@mail.gmail.com> I have the ?following problem. Could you tell me the solution? Executing the job in my cluster, Carpet create several threads automatically. For example, when I use 4 node( 2 core per node), i.e., the number of total core are 8, Carpet create 16 threads like this: INFO (Carpet): Carpet is running on 8 processes INFO (Carpet): This is process 0 INFO (Carpet): This process contains 2 threads INFO (Carpet): There are 16 threads in total Therefore I want to control the number of threads, i.e., I want to use only 8 threads in this case. This become more serious in the computer with a lot of cores. In the case of our workstation with 8 core, the number of threads are 64!!! like this: INFO (Carpet): Carpet is running on 8 processes INFO (Carpet): This is process 0 INFO (Carpet): This process contains 8 threads INFO (Carpet): There are 64 threads in total and calculation speed become very slow... ----------------------- Kentaro Takami ----------------------- From knarf at cct.lsu.edu Thu Aug 6 11:30:29 2009 From: knarf at cct.lsu.edu (Frank Loeffler) Date: Thu, 6 Aug 2009 11:30:29 -0500 Subject: [Users] How to control the number of threads? In-Reply-To: <4c323ec00908060923n78d591ecy9d00904e8b1fd965@mail.gmail.com> References: <4c323ec00908060923n78d591ecy9d00904e8b1fd965@mail.gmail.com> Message-ID: <20090806163028.GW29820@numrel07.cct.lsu.edu> On Fri, Aug 07, 2009 at 01:23:04AM +0900, KENTARO TAKAMI wrote: > I have the ?following problem. > Could you tell me the solution? Could this be an openmp problem? Please try setting the environment variable OMP_NUM_THREADS=1 before running the executable. Sometimes you have to do that in the mpirun command itself (e.g. mpirun /usr/bin/env OMP_NUM_THREADS=1), but often it is enough to set it in the submission script before mpirun is called. If this variable is not set and the binary is compiled with openmp, Carpet will use as much openmp threads per mpi process as the openmp library will start - which is usually the number of available cores. Of course that is not what you want if you already have one mpi process per core. Alternativly if your code can utilize openmp, you should probably use less mpi processes per node. Another possibility would be to compile without openmp flags. Frank From takami at theo.phys.sci.hiroshima-u.ac.jp Thu Aug 6 12:48:53 2009 From: takami at theo.phys.sci.hiroshima-u.ac.jp (KENTARO TAKAMI) Date: Fri, 7 Aug 2009 02:48:53 +0900 Subject: [Users] How to control the number of threads? In-Reply-To: <20090806163028.GW29820@numrel07.cct.lsu.edu> References: <4c323ec00908060923n78d591ecy9d00904e8b1fd965@mail.gmail.com> <20090806163028.GW29820@numrel07.cct.lsu.edu> Message-ID: <4c323ec00908061048r596c0b85q71881615f7c657ed@mail.gmail.com> Thank you for prompt reply. > Please try setting the environment > variable OMP_NUM_THREADS=1 before running the executable. That's right. My problem was solved by your advice. Thank you. ----------------------- Kentaro Takami ----------------------- From dprideout at gmail.com Thu Aug 6 14:46:07 2009 From: dprideout at gmail.com (David Rideout) Date: Thu, 6 Aug 2009 15:46:07 -0400 Subject: [Users] large integer types Message-ID: <1ce81abb0908061246y50931b57i3a4577860d92809c@mail.gmail.com> Does anyone know of a machine which supports a >8 byte integer type? Thanks, David From schnetter at cct.lsu.edu Thu Aug 6 17:46:01 2009 From: schnetter at cct.lsu.edu (Erik Schnetter) Date: Thu, 6 Aug 2009 23:46:01 +0100 Subject: [Users] initialize parameters for HDF5 dataset In-Reply-To: <1e8c69dc0908060800p1d0750cav7b5aaf4b102c8725@mail.gmail.com> References: <1e8c69dc0908050640q1118c899jf6de6aba0c47a7e1@mail.gmail.com> <54E0F6D1-F6EC-45BF-8EFE-C3C53C58DF2B@cct.lsu.edu> <1e8c69dc0908050902o4fd867cbi9d0589343a31fd4d@mail.gmail.com> <937F13EE-5A53-408C-BA9A-0CD8DD40CF0B@cct.lsu.edu> <1e8c69dc0908060800p1d0750cav7b5aaf4b102c8725@mail.gmail.com> Message-ID: <0BF39A7D-5159-4204-AD06-E0610C9298A5@cct.lsu.edu> On Aug 6, 2009, at 16:00:57, Yin Feng wrote: > I followed instructions to add somethings in my .par file: > > IO::filereader_ID_dir = "/work/yfeng1/BlackOilSim/SPE11/datafiles/" > IO::filereader_ID_files = "Pw.h5" > IO::filereader_ID_vars = "BlackOilBase::Pressure" > > where, Pw.h5 is the HDF5 file only contain parameter Pw > Pressure is the group name which includes Pw, Po, and Pg > > when I try to make it run, it shows > INFO (IOUtil): Reading variables from data file 'Pw.h5' > WARNING[L1,P2] (IOUtil): Failed to read variables from data file > 'Pw.h5' > > So, is there any thing missing in my .par file? Yin The screen output indicates that Cactus tried to read variables from the HDF5 file but failed. How did you produce this file? Is it in Cactus HDF5 format? Pure HDF5 is a rather low-level format, it lacks e.g. information about the coordinates associated with each data point. Cactus adds its own attributes to define these. -erik -- Erik Schnetter http://www.cct.lsu.edu/~eschnett/ My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from . -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 203 bytes Desc: This is a digitally signed message part Url : http://www.cactuscode.org/pipermail/users/attachments/20090806/d699dcbb/attachment.bin From yfeng1 at tigers.lsu.edu Thu Aug 6 22:23:08 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Thu, 6 Aug 2009 22:23:08 -0500 Subject: [Users] initialize parameters for HDF5 dataset In-Reply-To: <0BF39A7D-5159-4204-AD06-E0610C9298A5@cct.lsu.edu> References: <1e8c69dc0908050640q1118c899jf6de6aba0c47a7e1@mail.gmail.com> <54E0F6D1-F6EC-45BF-8EFE-C3C53C58DF2B@cct.lsu.edu> <1e8c69dc0908050902o4fd867cbi9d0589343a31fd4d@mail.gmail.com> <937F13EE-5A53-408C-BA9A-0CD8DD40CF0B@cct.lsu.edu> <1e8c69dc0908060800p1d0750cav7b5aaf4b102c8725@mail.gmail.com> <0BF39A7D-5159-4204-AD06-E0610C9298A5@cct.lsu.edu> Message-ID: <1e8c69dc0908062023i5f8e26ag15b1207072a1aad9@mail.gmail.com> My output are like following: iohdf5::out_every = 200 iohdf5::out_dir = "/work/yfeng1/BlackOilSim/SPE11/SPE11" iohdf5::out_vars = "blackoilbase::Pw blackoilbase::Qo " IO::out_mode = "onefile" IO::out_unchunked = "yes" Is there any problem? Thank you, Erik! On Thu, Aug 6, 2009 at 5:46 PM, Erik Schnetter wrote: > On Aug 6, 2009, at 16:00:57, Yin Feng wrote: > > I followed instructions to add somethings in my .par file: >> >> IO::filereader_ID_dir = "/work/yfeng1/BlackOilSim/SPE11/datafiles/" >> IO::filereader_ID_files = "Pw.h5" >> IO::filereader_ID_vars = "BlackOilBase::Pressure" >> >> where, Pw.h5 is the HDF5 file only contain parameter Pw >> Pressure is the group name which includes Pw, Po, and Pg >> >> when I try to make it run, it shows >> INFO (IOUtil): Reading variables from data file 'Pw.h5' >> WARNING[L1,P2] (IOUtil): Failed to read variables from data file 'Pw.h5' >> >> So, is there any thing missing in my .par file? >> > > > Yin > > The screen output indicates that Cactus tried to read variables from the > HDF5 file but failed. How did you produce this file? Is it in Cactus HDF5 > format? > > Pure HDF5 is a rather low-level format, it lacks e.g. information about the > coordinates associated with each data point. Cactus adds its own attributes > to define these. > > > -erik > > -- > Erik Schnetter http://www.cct.lsu.edu/~eschnett/ > > My email is as private as my paper mail. I therefore support encrypting > and signing email messages. Get my PGP key from . > > > > > _______________________________________________ > Users mailing list > Users at cactuscode.org > http://www.cactuscode.org/mailman/listinfo/users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.cactuscode.org/pipermail/users/attachments/20090806/2ed9b9f8/attachment.html From schnetter at cct.lsu.edu Mon Aug 10 08:04:02 2009 From: schnetter at cct.lsu.edu (Erik Schnetter) Date: Mon, 10 Aug 2009 15:04:02 +0200 Subject: [Users] large integer types In-Reply-To: <1ce81abb0908061246y50931b57i3a4577860d92809c@mail.gmail.com> References: <1ce81abb0908061246y50931b57i3a4577860d92809c@mail.gmail.com> Message-ID: On Aug 6, 2009, at 20:46:07, David Rideout wrote: > Does anyone know of a machine which supports a >8 byte integer type? I am not aware of a machine that supports this. There are, of course, libraries for large integer calculations, but these are slower than regular integers. You could also use quad precision floating point numbers (if they exist), as integer arithmetic using floating point numbers is required to be exact by the IEEE standard. -erik -- Erik Schnetter http://www.cct.lsu.edu/~eschnett/ My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from . -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 203 bytes Desc: This is a digitally signed message part Url : http://www.cactuscode.org/pipermail/users/attachments/20090810/aa256543/attachment.bin From jshalf at lbl.gov Mon Aug 10 09:35:15 2009 From: jshalf at lbl.gov (John Shalf) Date: Mon, 10 Aug 2009 07:35:15 -0700 Subject: [Users] large integer types In-Reply-To: References: <1ce81abb0908061246y50931b57i3a4577860d92809c@mail.gmail.com> Message-ID: <968BF237-346F-4181-B538-96BF1166D2A8@lbl.gov> There is a GNU software library for arbitrary precision Integer datatypes. It is used for cryptoanalysis. There is no direct machine support for this, but the emulation is pretty good. http://gmplib.org/ -john On Aug 10, 2009, at 6:04 AM, Erik Schnetter wrote: > On Aug 6, 2009, at 20:46:07, David Rideout wrote: > >> Does anyone know of a machine which supports a >8 byte integer type? > > > I am not aware of a machine that supports this. > > There are, of course, libraries for large integer calculations, but > these are slower than regular integers. You could also use quad > precision floating point numbers (if they exist), as integer > arithmetic using floating point numbers is required to be exact by > the IEEE standard. > > -erik > > -- > Erik Schnetter http://www.cct.lsu.edu/~eschnett/ > > My email is as private as my paper mail. I therefore support > encrypting > and signing email messages. Get my PGP key from . > > > > _______________________________________________ > Users mailing list > Users at cactuscode.org > http://www.cactuscode.org/mailman/listinfo/users From dprideout at gmail.com Tue Aug 11 11:34:08 2009 From: dprideout at gmail.com (David Rideout) Date: Tue, 11 Aug 2009 12:34:08 -0400 Subject: [Users] large integer types In-Reply-To: <968BF237-346F-4181-B538-96BF1166D2A8@lbl.gov> References: <1ce81abb0908061246y50931b57i3a4577860d92809c@mail.gmail.com> <968BF237-346F-4181-B538-96BF1166D2A8@lbl.gov> Message-ID: <1ce81abb0908110934x622063a7mfa8da760ddcfb1ef@mail.gmail.com> Interesting, thanks! -David On Mon, Aug 10, 2009 at 10:35 AM, John Shalf wrote: > > There is a GNU software library for arbitrary precision Integer > datatypes. It is used for cryptoanalysis. > There is no direct machine support for this, but the emulation is > pretty good. > > http://gmplib.org/ > > -john > > On Aug 10, 2009, at 6:04 AM, Erik Schnetter wrote: >> On Aug 6, 2009, at 20:46:07, David Rideout wrote: >> >>> Does anyone know of a machine which supports a >8 byte integer type? >> >> >> I am not aware of a machine that supports this. >> >> There are, of course, libraries for large integer calculations, but >> these are slower than regular integers. You could also use quad >> precision floating point numbers (if they exist), as integer >> arithmetic using floating point numbers is required to be exact by >> the IEEE standard. >> >> -erik >> >> -- >> Erik Schnetter http://www.cct.lsu.edu/~eschnett/ >> >> My email is as private as my paper mail. I therefore support >> encrypting >> and signing email messages. Get my PGP key from . >> >> >> >> _______________________________________________ >> Users mailing list >> Users at cactuscode.org >> http://www.cactuscode.org/mailman/listinfo/users > > _______________________________________________ > Users mailing list > Users at cactuscode.org > http://www.cactuscode.org/mailman/listinfo/users > From knarf at cct.lsu.edu Wed Aug 12 13:41:04 2009 From: knarf at cct.lsu.edu (Frank Loeffler) Date: Wed, 12 Aug 2009 13:41:04 -0500 Subject: [Users] initialize parameters for HDF5 dataset In-Reply-To: <1e8c69dc0908060800p1d0750cav7b5aaf4b102c8725@mail.gmail.com> References: <1e8c69dc0908050640q1118c899jf6de6aba0c47a7e1@mail.gmail.com> <54E0F6D1-F6EC-45BF-8EFE-C3C53C58DF2B@cct.lsu.edu> <1e8c69dc0908050902o4fd867cbi9d0589343a31fd4d@mail.gmail.com> <937F13EE-5A53-408C-BA9A-0CD8DD40CF0B@cct.lsu.edu> <1e8c69dc0908060800p1d0750cav7b5aaf4b102c8725@mail.gmail.com> Message-ID: <20090812184103.GZ24677@numrel07.cct.lsu.edu> Hi, for the sake of documenting the resolution of this, the following line was the problem: > IO::filereader_ID_files = "Pw.h5" What you would have to give here is the 'base file name', e.g. without the extention ".h5": IO::filereader_ID_files = "Pw" However, both the description of the parameter and the error messages suggested otherwise: > INFO (IOUtil): Reading variables from data file 'Pw.h5' > WARNING[L1,P2] (IOUtil): Failed to read variables from data file 'Pw.h5' The description of the parameter and both messages are now changed to reflect that issue with the 'base file name' better in the hope that this can be catched easier in the future. Frank From yfeng1 at tigers.lsu.edu Wed Aug 12 16:00:59 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Wed, 12 Aug 2009 16:00:59 -0500 Subject: [Users] initialize parameters for HDF5 dataset In-Reply-To: <20090812184103.GZ24677@numrel07.cct.lsu.edu> References: <1e8c69dc0908050640q1118c899jf6de6aba0c47a7e1@mail.gmail.com> <54E0F6D1-F6EC-45BF-8EFE-C3C53C58DF2B@cct.lsu.edu> <1e8c69dc0908050902o4fd867cbi9d0589343a31fd4d@mail.gmail.com> <937F13EE-5A53-408C-BA9A-0CD8DD40CF0B@cct.lsu.edu> <1e8c69dc0908060800p1d0750cav7b5aaf4b102c8725@mail.gmail.com> <20090812184103.GZ24677@numrel07.cct.lsu.edu> Message-ID: <1e8c69dc0908121400v11da46d5pc6232a4ccc5b84b3@mail.gmail.com> Hi, Frank When I try using the function IOUtil_RecoverVarsFromDatafiles(cctkGH,"/work/yfeng1/datafiles/Porosity","BlackOilBase::Porosity"), Cactus will add "./" at the beginning the datafile path string "/work/yfeng1/datafiles/Porosity", and it becomes ".//work/yfeng1/datafiles/Porosity". That means, what I input is an absolute path, but Cactus changes it to a relative path. For parallel computation, I need absolute path to locate resources. So, is there any way to avoid Cactus making changes to the path I input? Thank you! On Wed, Aug 12, 2009 at 1:41 PM, Frank Loeffler wrote: > Hi, > > for the sake of documenting the resolution of this, the following line > was the problem: > > > IO::filereader_ID_files = "Pw.h5" > > What you would have to give here is the 'base file name', e.g. without > the extention ".h5": IO::filereader_ID_files = "Pw" > > However, both the description of the parameter and the error messages > suggested otherwise: > > > INFO (IOUtil): Reading variables from data file 'Pw.h5' > > WARNING[L1,P2] (IOUtil): Failed to read variables from data file 'Pw.h5' > > The description of the parameter and both messages are now changed to > reflect that issue with the 'base file name' better in the hope that > this can be catched easier in the future. > > Frank > > _______________________________________________ > Users mailing list > Users at cactuscode.org > http://www.cactuscode.org/mailman/listinfo/users > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.cactuscode.org/pipermail/users/attachments/20090812/bf7db93d/attachment.html From takami at theo.phys.sci.hiroshima-u.ac.jp Fri Aug 21 04:43:11 2009 From: takami at theo.phys.sci.hiroshima-u.ac.jp (KENTARO TAKAMI) Date: Fri, 21 Aug 2009 18:43:11 +0900 Subject: [Users] Job is aborted in my cluster. Message-ID: <4c323ec00908210243l197b0bc3jb7a876a613d87be7@mail.gmail.com> Hi, I encounter a trouble, but I can't understand a cause of it. Could you give me advice about it? In certain cluster which consist of several personal computer, when I execute the job, the job is aborted after several iteration. Every time, the aborted time is different, that is, for certain case, tens of iteration is completed, and for certain case, hundreds of iteration is completed. (In other workstation, the same job is not aborted.) The error log is the following. (Is error signal induced by CarpetReduce?) <9:0>SCore: *** SIGNAL EXCEPTION eip=0x088019b0, cr2=0x 0 *** <9:0>SCore: gs=0x0000, fs=0x0000, es=0x002b, ds=0x002b <9:0>SCore: edi=0x0cd6b498, esi=0x00000000, ebp=0xbfffa938, esp=0xbfffa880 <9:0>SCore: ebx=0xbfffa978, edx=0xbfffab0c, ecx=0x00005c48, eax=0x0ccb45b8 <9:0>SCore: trapno=0x00000013, err=0x00000000, eip=0x088019b0, cs=0x0023 <9:0>SCore: esp_at_signal=0xbfffa880, ss=0x002b, oldmask=0x80000000,cr2=0x00000000 <9:0> Trying to attach GDB (no DISPLAY): Exception signal (SIGFPE) <9:0> GDB invoked by SCore (batch) 0x08cbee4d in syscall () #0 0x08cbee4d in syscall () #1 0x08b3a474 in msync_caller () at ../scwrap.c:257 #2 0x08b3a2b1 in vsyscall (handle=0xbfffa2fc, retp=0xbfffa2d8, args=0xbfffa2cc "?????????\224??????????\b??????") at ../scwrap.c:748 #3 0x08b393e5 in score_syscall (handle=0x9578e60, retp=0xbfffa2d8) at ../scwrap.c:763 #4 0x08b39ae6 in __nanosleep (req=0xbfffa2fc, rem=0xbfffa2fc) at ../scwrap.c:1201 #5 0x08cacac3 in sleep () #6 0x08b2b8e6 in score_wait_forever () at ../libsc_util.c:154 #7 0x08b2bdf5 in sc_inspectme (x_display=0x0, signal=-4) at ../libscio.c:243 #8 0x08b239f9 in MPID_SCORE_Exception () #9 #10 0x088019b0 in CarpetReduce::Reduce(_cGH const*, int, int const*, int const*, int const*, int, std::vector > const&, std::vector > const&, int, int, void*, int, void*, CarpetReduce::reduction const*, double const*, double) () -- ----------------------- Kentaro Takami ----------------------- From schnetter at cct.lsu.edu Fri Aug 21 07:01:49 2009 From: schnetter at cct.lsu.edu (Erik Schnetter) Date: Fri, 21 Aug 2009 14:01:49 +0200 Subject: [Users] Job is aborted in my cluster. In-Reply-To: <4c323ec00908210243l197b0bc3jb7a876a613d87be7@mail.gmail.com> References: <4c323ec00908210243l197b0bc3jb7a876a613d87be7@mail.gmail.com> Message-ID: <3E740AB0-63EB-4580-B9C0-136AB14E8FA6@cct.lsu.edu> On Aug 21, 2009, at 11:43:11, KENTARO TAKAMI wrote: > Hi, > > I encounter a trouble, but I can't understand a cause of it. > Could you give me advice about it? > > In certain cluster which consist of several personal computer, > when I execute the job, the job is aborted after several iteration. > > Every time, the aborted time is different, that is, > for certain case, tens of iteration is completed, > and for certain case, hundreds of iteration is completed. > (In other workstation, the same job is not aborted.) > > The error log is the following. (Is error signal induced by > CarpetReduce?) Kentaro From the error message it looks as if this was a floating point exception. A floating point exception is caused (depending on system settings) when a mathematically undefined operation is performed, or when there is an integer division by zero. Since the error occurs at a different place each time, it is likely that the error is caused by a hardware problem, not a software problem. This also fits the evidence that another workstation does not show this error. Hardware errors can be caused e.g. by overheating or by faulty memory. Tracking down and confirming hardware errors can be difficult. If this is really a hardware problem, then you could try to run the simulation on N-1 computers instead of N computers, and in this way try to find out which of the computers is causing the problem. -erik -- Erik Schnetter http://www.cct.lsu.edu/~eschnett/ My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from . -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 203 bytes Desc: This is a digitally signed message part Url : http://www.cactuscode.org/pipermail/users/attachments/20090821/3c182964/attachment.bin From takami at theo.phys.sci.hiroshima-u.ac.jp Sun Aug 23 14:55:45 2009 From: takami at theo.phys.sci.hiroshima-u.ac.jp (KENTARO TAKAMI) Date: Mon, 24 Aug 2009 04:55:45 +0900 Subject: [Users] Job is aborted in my cluster. In-Reply-To: <3E740AB0-63EB-4580-B9C0-136AB14E8FA6@cct.lsu.edu> References: <4c323ec00908210243l197b0bc3jb7a876a613d87be7@mail.gmail.com> <3E740AB0-63EB-4580-B9C0-136AB14E8FA6@cct.lsu.edu> Message-ID: <4c323ec00908231255s7a77f355l29ba7bff6d16e96f@mail.gmail.com> Hi, Erik, Thank you for advice. > If this is really a hardware problem, then you could try to run the > simulation on N-1 computers instead of N computers, and in this way try to > find out which of the computers is causing the problem. I tried several test and I could avoid my problem. My cluster and OpenMP seem to be incompatible. Therefore, if I use "OPENMP = no" in compiling, job is never aborted. -- ----------------------- Kentaro Takami ----------------------- From schnetter at cct.lsu.edu Mon Aug 24 02:30:16 2009 From: schnetter at cct.lsu.edu (Erik Schnetter) Date: Mon, 24 Aug 2009 09:30:16 +0200 Subject: [Users] Job is aborted in my cluster. In-Reply-To: <4c323ec00908231255s7a77f355l29ba7bff6d16e96f@mail.gmail.com> References: <4c323ec00908210243l197b0bc3jb7a876a613d87be7@mail.gmail.com> <3E740AB0-63EB-4580-B9C0-136AB14E8FA6@cct.lsu.edu> <4c323ec00908231255s7a77f355l29ba7bff6d16e96f@mail.gmail.com> Message-ID: <9EB9718B-FABA-43E1-A06A-98DE1853280A@cct.lsu.edu> On Aug 23, 2009, at 21:55:45, KENTARO TAKAMI wrote: > Hi, Erik, > > Thank you for advice. > >> If this is really a hardware problem, then you could try to run the >> simulation on N-1 computers instead of N computers, and in this way >> try to >> find out which of the computers is causing the problem. > > I tried several test and I could avoid my problem. > My cluster and OpenMP seem to be incompatible. > Therefore, if I use "OPENMP = no" in compiling, job is never aborted. Congratulations for finding the problem! -erik -- Erik Schnetter http://www.cct.lsu.edu/~eschnett/ My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from . -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 203 bytes Desc: This is a digitally signed message part Url : http://www.cactuscode.org/pipermail/users/attachments/20090824/ea3b0f5e/attachment.bin From yfeng1 at tigers.lsu.edu Sat Aug 29 22:44:10 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Sat, 29 Aug 2009 22:44:10 -0500 Subject: [Users] Cactus timing Message-ID: <1e8c69dc0908292044r1b8edc09yb28de25ac04ee5a@mail.gmail.com> Does anyone know whether Cactus provide timing functions for parallel computation? The function may like following in MPI: double starttime, endtime; starttime = MPI_Wtime(); endtime = MPI_Wtime(); printf("wall clock time = %f\n", endtime-starttime); Thank you in advance! Sincerely, YIN -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.cactuscode.org/pipermail/users/attachments/20090829/e3f41007/attachment.html From schnetter at cct.lsu.edu Sun Aug 30 08:13:45 2009 From: schnetter at cct.lsu.edu (Erik Schnetter) Date: Sun, 30 Aug 2009 08:13:45 -0500 Subject: [Users] Cactus timing In-Reply-To: <1e8c69dc0908292044r1b8edc09yb28de25ac04ee5a@mail.gmail.com> References: <1e8c69dc0908292044r1b8edc09yb28de25ac04ee5a@mail.gmail.com> Message-ID: <45B82CFD-5D98-4B2F-ADB4-134F461908E8@cct.lsu.edu> On Aug 29, 2009, at 10:44 PM, Yin Feng wrote: > Does anyone know whether Cactus provide timing functions for > parallel computation? Yes, Cactus has many built-in timers and clocks. You create, start, and stop timers with: int itimer = CCTK_TimerCreate ("SomeUsefulName"); CCTK_TimerStartI (itimer); CCTK_TimerStop( (itimer); Each timer has several clocks that can measure different things, e.g. wall time and CPU time. There are also functions to print timers and clocks, but the easiest way is to use the thorn TimerReport which periodically outputs all timers to screen or to a file. (I especially recommend to set TimerReport::n_top_timers=20, which will output the names of the 20 most expensive routines.) You will notice that Cactus automatically places timers around all scheduled routines. The users' guide and the reference manuals have a sections on timers with more detail. -erik -- Erik Schnetter http://www.cct.lsu.edu/~eschnett/ My email is as private as my paper mail. I therefore support encrypting and signing email messages. Get my PGP key from . -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 203 bytes Desc: This is a digitally signed message part Url : http://www.cactuscode.org/pipermail/users/attachments/20090830/dbf1c137/attachment.bin From yfeng1 at tigers.lsu.edu Sun Aug 30 16:36:53 2009 From: yfeng1 at tigers.lsu.edu (Yin Feng) Date: Sun, 30 Aug 2009 16:36:53 -0500 Subject: [Users] An error received when I run Cactus application Message-ID: <1e8c69dc0908301436u75e2110by54858ff4c141f9cc@mail.gmail.com> I have a cactus application it runs properly on Queenbee, and when I run the same program on tezpur (I compiled it with same settings), I got following errors: Sorry! You were supposed to get help about: opal_init:startup:internal-failure from the file: help-opal-runtime.txt But I couldn't find any file matching that name. Sorry! -------------------------------------------------------------------------- [tezpur316:28045] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file runtime/orte_init.c at line 77 [tezpur316:28045] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file orterun.c at line 454 Does anyone know what's happening Thank you in advance! Sincerely, YIN -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.cactuscode.org/pipermail/users/attachments/20090830/ddca152c/attachment.html