# 大网格-奇怪报错

• 还是在算圆柱绕流。为了进入critical雷诺数，网格加密了很多，导致的问题也很多。192核那个例子的网格数在600W，运行没有什么问题。

可是在其他条件相同，但是用更多的网格的288核算例中出现奇怪的错误。读取初始和边界条件，湍流模型都没有什么问题，就是到 smoothsovler 那里出现了问题。

288核这个case，网格数在3千万。

个人感觉可能是内存的问题，但是我们这的计算单个节点内存是32GB，照理说12个节点的总内存量应该是能应付的？难道是计算到smoothsovler这一步，在某一个节点的内存量大大超过了32GB？

192 正常运算

Create time

Create mesh for time = 300

PISO: Operating solver in PISO mode

Selecting incompressible transport model Newtonian
Selecting turbulence model type LES
Selecting LES turbulence model dynamicKEqn
Selecting LES delta type cubeRootVol
bounding k, min: 0 max: 0.0015093 average: 1.99878e-05
dynamicKEqnCoeffs
{
filter          simple;
Ce              1.048;
}

No MRF models present

No finite volume options present

Time = 300.02

Courant Number mean: 0.026086 max: 0.983623
smoothSolver:  Solving for Ux, Initial residual = 0.000816211, Final residual = 2.84247e-08,\
No Iterations 3
smoothSolver:  Solving for Uy, Initial residual = 0.00267408, Final residual = 2.09581e-08, \
No Iterations 4
smoothSolver:  Solving for Uz, Initial residual = 0.00457375, Final residual = 3.57969e-08, \
No Iterations 4
GAMG:  Solving for p, Initial residual = 0.220597, Final residual = 0.00856105, No Iteration\
s 3
time step continuity errors : sum local = 6.24882e-08, global = 5.53688e-10, cumulative = 5.\
53688e-10
GAMG:  Solving for p, Initial residual = 0.00891699, Final residual = 0.000438876, No Iterat\
ions 12
time step continuity errors : sum local = 3.18791e-09, global = -1.1464e-10, cumulative = 4.\
39048e-10
GAMG:  Solving for p, Initial residual = 0.000445623, Final residual = 9.53731e-08, No Itera\
tions 25
time step continuity errors : sum local = 6.89127e-13, global = -7.61126e-14, cumulative = 4\
.38972e-10
smoothSolver:  Solving for k, Initial residual = 0.000553063, Final residual = 3.40938e-08, \
No Iterations 3
bounding k, min: -3.68878e-09 max: 0.00149781 average: 1.99956e-05
ExecutionTime = 6.64 s  ClockTime = 7 s



288 核 异常错误

Create time

Create mesh for time = 0

PISO: Operating solver in PISO mode

Selecting incompressible transport model Newtonian
Selecting turbulence model type LES
Selecting LES turbulence model dynamicKEqn
Selecting LES delta type cubeRootVol
bounding k, min: 0 max: 2e-05 average: 0
dynamicKEqnCoeffs
{
filter          simple;
Ce              1.048;
}

No MRF models present

No finite volume options present

Starting time loop

Time = 0.005

Courant Number mean: 0.000119441 max: 0.017568
smoothSolver:  Solving for Ux, Initial residual = 1, Final residual = 2.06726e-09, No Iterat\
ions 2
smoothSolver:  Solving for Uy, Initial residual = 1, Final residual = 1.30712e-09, No Iterat\
ions 2
smoothSolver:  Solving for Uz, Initial residual = 1, Final residual = 1.31187e-09, No Iterat\
ions 2
GAMG:  Solving for p, Initial residual = 1, Final residual = 0.035832, No Iterations 36
time step continuity errors : sum local = 8.55925e-06, global = -9.25811e-08, cumulative = -\
9.25811e-08
GAMG:  Solving for p, Initial residual = 0.000128631, Final residual = 6.05672e-06, No Itera\
tions 39
time step continuity errors : sum local = 4.27446e-07, global = -5.85326e-09, cumulative = -\
9.84343e-08
GAMG:  Solving for p, Initial residual = 8.87864e-06, Final residual = 9.28164e-08, No Itera\
tions 74
time step continuity errors : sum local = 6.57138e-09, global = 9.95505e-13, cumulative = -9\
.84333e-08
--------------------------------------------------------------------------
mpiexec noticed that process rank 105 with PID 0 on node cp1056 exited on signal 8 (Floating\
point exception).
--------------------------------------------------------------------------


另外的报错信息：

An MPI process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your MPI job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

Local host:          cp1815 (PID 31613)
MPI_COMM_WORLD rank: 89

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec noticed that process rank 105 with PID 0 on node cp1652 exited on signal 8 (Floating\
point exception).


我查找了相关信息，在提交作业的脚本文件做了相应的改变：

mpiexec --mca mpi_warn_on_fork 0 -n 288 pisoFoam -parallel > simpleFoam_${PBS_JOBID}.out  还是不能计算 Time = 0.005 Courant Number mean: 0.000119441 max: 0.017568 smoothSolver: Solving for Ux, Initial residual = 1, Final residual = 2.06726e-09, No Iterat\ ions 2 smoothSolver: Solving for Uy, Initial residual = 1, Final residual = 1.30712e-09, No Iterat\ ions 2 smoothSolver: Solving for Uz, Initial residual = 1, Final residual = 1.31187e-09, No Iterat\ ions 2 GAMG: Solving for p, Initial residual = 1, Final residual = 0.035832, No Iterations 36 time step continuity errors : sum local = 8.55925e-06, global = -9.25811e-08, cumulative = -\ 9.25811e-08 GAMG: Solving for p, Initial residual = 0.000128631, Final residual = 6.05672e-06, No Itera\ tions 39 time step continuity errors : sum local = 4.27446e-07, global = -5.85326e-09, cumulative = -\ 9.84343e-08 GAMG: Solving for p, Initial residual = 8.87864e-06, Final residual = 9.28164e-08, No Itera\ tions 74 time step continuity errors : sum local = 6.57138e-09, global = 9.95505e-13, cumulative = -9\ .84333e-08 -------------------------------------------------------------------------- mpiexec noticed that process rank 105 with PID 0 on node cp1056 exited on signal 8 (Floating\ point exception). --------------------------------------------------------------------------  还请有经验的朋友分析一下这个问题。 • @random_ran Sorry, Chinese input missing. mpiexec noticed that process rank 105 with PID 0 on node cp1056 exited on signal 8 (Floating\ point exception).  This is a quite general error, which does not mean it comes from setting up of mpi. Meanwhile, sometimes your simulation blows up in parallel running, see: link text. Which version of OpenFOAM do you use? Could u please try PBiCG instead of smoothSolver? How do you decompose your grid? • @cfd-china 补充信息： OpenFOAM v4.1  /*--------------------------------*- C++ -*----------------------------------*\ | ========= | | | \\ / F ield | OpenFOAM: The Open Source CFD Toolbox | | \\ / O peration | Version: 4.x | | \\ / A nd | Web: www.OpenFOAM.org | | \\/ M anipulation | | \*---------------------------------------------------------------------------*/ FoamFile { version 2.0; format ascii; class dictionary; location "system"; object decomposeParDict; } // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * // numberOfSubdomains 288; method hierarchical; hierarchicalCoeffs { n (16 9 2); delta 0.001; order xyz; hierarchicalCoeffs { n (16 9 2); delta 0.001; order xyz; } manualCoeffs { dataFile "cellDecomposition"; } // ************************************************************************* //  *--------------------------------*- C++ -*----------------------------------*\ | ========= | | | \\ / F ield | OpenFOAM: The Open Source CFD Toolbox | | \\ / O peration | Version: 4.x | | \\ / A nd | Web: www.OpenFOAM.org | | \\/ M anipulation | | \*---------------------------------------------------------------------------*/ FoamFile { version 2.0; format ascii; class dictionary; location "system"; object fvSolution; } // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * // solvers { p { solver GAMG; tolerance 1e-07; relTol 0.05; smoother GaussSeidel; } pFinal {$p;
smoother        DICGaussSeidel;
tolerance       1e-07;
relTol          0;
}

"(U|k|B|nuTilda|s)"
{
solver          smoothSolver;
smoother        GaussSeidel;
tolerance       1e-07;
relTol          0;
}
}

PISO
{
nCorrectors     3;
nNonOrthogonalCorrectors 0;
}

relaxationFactors
{
"U.*"               1;
"nuTilda.*"         1;
}



• 再补充：

192 case 的雷诺数在10^4
288 case 的在10^5

这两个case的分解方法，controDict, fvSchemes, fvSolution 基本一样。
不同的是288 case 的时间步长更小而已。

• k 文件 tubulent kinetic energy：

*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  4.x                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
version     2.0;
format      ascii;
class       volScalarField;
object      k;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

dimensions      [0 2 -2 0 0 0 0];

internalField   uniform 0;

boundaryField
{
INLET
{
type            fixedValue;
value           uniform 2e-05;
}

OUTLET
{
type            inletOutlet;
inletValue      uniform 0;
value           uniform 0;
}

BACK_CYC
{
type            cyclic;
}
FRONT_CYC
{
type            cyclic;
}

CYLINDER
{
type            fixedValue;
value           uniform 0;
}

}


• @cfd-china

非常感谢你的建议！非常管用，起码现在能跑起来了。


Time = 0.005

Courant Number mean: 0.000119441 max: 0.017568
DILUPBiCG:  Solving for Ux, Initial residual = 1, Final residual = 2.34151e-11, No Iterations 2
DILUPBiCG:  Solving for Uy, Initial residual = 1, Final residual = 1.61665e-10, No Iterations 2
DILUPBiCG:  Solving for Uz, Initial residual = 1, Final residual = 1.63882e-10, No Iterations 2
GAMG:  Solving for p, Initial residual = 1, Final residual = 0.035832, No Iterations 36
time step continuity errors : sum local = 8.55925e-06, global = -9.25811e-08, cumulative = -9.25811e-08
GAMG:  Solving for p, Initial residual = 0.000128631, Final residual = 6.05672e-06, No Iterations 39
time step continuity errors : sum local = 4.27446e-07, global = -5.85326e-09, cumulative = -9.84343e-08
GAMG:  Solving for p, Initial residual = 8.87864e-06, Final residual = 9.28164e-08, No Iterations 74
time step continuity errors : sum local = 6.57138e-09, global = 9.95505e-13, cumulative = -9.84333e-08
DILUPBiCG:  Solving for k, Initial residual = 0.297752, Final residual = 7.56559e-07, No Iterations 4
bounding k, min: -2.24189e-08 max: 2e-05 average: 2.54114e-09
ExecutionTime = 42.97 s  ClockTime = 43 s


我改动的fvSolution 部分是：

    "(U|k|B|nuTilda|s)"
{
solver          PBiCG;
preconditioner  DILU
tolerance       1e-07;
relTol          0;
}


目前运行了半个小时了。

我知道后面还会有的新的问题。