大网格-奇怪报错


  • 版主

    还是在算圆柱绕流。为了进入critical雷诺数,网格加密了很多,导致的问题也很多。192核那个例子的网格数在600W,运行没有什么问题。

    可是在其他条件相同,但是用更多的网格的288核算例中出现奇怪的错误。读取初始和边界条件,湍流模型都没有什么问题,就是到 smoothsovler 那里出现了问题。

    288核这个case,网格数在3千万。

    个人感觉可能是内存的问题,但是我们这的计算单个节点内存是32GB,照理说12个节点的总内存量应该是能应付的?难道是计算到smoothsovler这一步,在某一个节点的内存量大大超过了32GB?

    192 正常运算

    Create time
    
    Create mesh for time = 300
    
    
    PISO: Operating solver in PISO mode
    
    Reading field p
    
    Reading field U
    
    Reading/calculating face flux field phi
    
    Selecting incompressible transport model Newtonian
    Selecting turbulence model type LES
    Selecting LES turbulence model dynamicKEqn
    Selecting LES delta type cubeRootVol
    bounding k, min: 0 max: 0.0015093 average: 1.99878e-05
    dynamicKEqnCoeffs
    {
        filter          simple;
        Ce              1.048;
    }
    
    No MRF models present
    
    No finite volume options present
    
    Time = 300.02
    
    Courant Number mean: 0.026086 max: 0.983623
    smoothSolver:  Solving for Ux, Initial residual = 0.000816211, Final residual = 2.84247e-08,\
     No Iterations 3
    smoothSolver:  Solving for Uy, Initial residual = 0.00267408, Final residual = 2.09581e-08, \
    No Iterations 4
    smoothSolver:  Solving for Uz, Initial residual = 0.00457375, Final residual = 3.57969e-08, \
    No Iterations 4
    GAMG:  Solving for p, Initial residual = 0.220597, Final residual = 0.00856105, No Iteration\
    s 3
    time step continuity errors : sum local = 6.24882e-08, global = 5.53688e-10, cumulative = 5.\
    53688e-10
    GAMG:  Solving for p, Initial residual = 0.00891699, Final residual = 0.000438876, No Iterat\
    ions 12
    time step continuity errors : sum local = 3.18791e-09, global = -1.1464e-10, cumulative = 4.\
    39048e-10
    GAMG:  Solving for p, Initial residual = 0.000445623, Final residual = 9.53731e-08, No Itera\
    tions 25
    time step continuity errors : sum local = 6.89127e-13, global = -7.61126e-14, cumulative = 4\
    .38972e-10
    smoothSolver:  Solving for k, Initial residual = 0.000553063, Final residual = 3.40938e-08, \
    No Iterations 3
    bounding k, min: -3.68878e-09 max: 0.00149781 average: 1.99956e-05
    ExecutionTime = 6.64 s  ClockTime = 7 s
    
    

    288 核 异常错误

    Create time
    
    Create mesh for time = 0
    
    
    PISO: Operating solver in PISO mode
    
    Reading field p
    
    Reading field U
    
    Reading/calculating face flux field phi
    
    Selecting incompressible transport model Newtonian
    Selecting turbulence model type LES
    Selecting LES turbulence model dynamicKEqn
    Selecting LES delta type cubeRootVol
    bounding k, min: 0 max: 2e-05 average: 0
    dynamicKEqnCoeffs
    {
        filter          simple;
        Ce              1.048;
    }
    
    No MRF models present
    
    No finite volume options present
    
    
    Starting time loop
    
    Time = 0.005
    
    
    Courant Number mean: 0.000119441 max: 0.017568
    smoothSolver:  Solving for Ux, Initial residual = 1, Final residual = 2.06726e-09, No Iterat\
    ions 2
    smoothSolver:  Solving for Uy, Initial residual = 1, Final residual = 1.30712e-09, No Iterat\
    ions 2
    smoothSolver:  Solving for Uz, Initial residual = 1, Final residual = 1.31187e-09, No Iterat\
    ions 2
    GAMG:  Solving for p, Initial residual = 1, Final residual = 0.035832, No Iterations 36
    time step continuity errors : sum local = 8.55925e-06, global = -9.25811e-08, cumulative = -\
    9.25811e-08
    GAMG:  Solving for p, Initial residual = 0.000128631, Final residual = 6.05672e-06, No Itera\
    tions 39
    time step continuity errors : sum local = 4.27446e-07, global = -5.85326e-09, cumulative = -\
    9.84343e-08
    GAMG:  Solving for p, Initial residual = 8.87864e-06, Final residual = 9.28164e-08, No Itera\
    tions 74
    time step continuity errors : sum local = 6.57138e-09, global = 9.95505e-13, cumulative = -9\
    .84333e-08
    --------------------------------------------------------------------------
    mpiexec noticed that process rank 105 with PID 0 on node cp1056 exited on signal 8 (Floating\
     point exception).
    --------------------------------------------------------------------------
    

    另外的报错信息:

    An MPI process has executed an operation involving a call to the
    "fork()" system call to create a child process.  Open MPI is currently
    operating in a condition that could result in memory corruption or
    other system errors; your MPI job may hang, crash, or produce silent
    data corruption.  The use of fork() (or system() or other calls that
    create child processes) is strongly discouraged.
    
    The process that invoked fork was:
    
      Local host:          cp1815 (PID 31613)
      MPI_COMM_WORLD rank: 89
    
    If you are *absolutely sure* that your application will successfully
    and correctly survive a call to fork(), you may disable this warning
    by setting the mpi_warn_on_fork MCA parameter to 0.
    --------------------------------------------------------------------------
    --------------------------------------------------------------------------
    mpiexec noticed that process rank 105 with PID 0 on node cp1652 exited on signal 8 (Floating\
     point exception).
    

    我查找了相关信息,在提交作业的脚本文件做了相应的改变:

    mpiexec --mca mpi_warn_on_fork 0 -n 288 pisoFoam -parallel > simpleFoam_${PBS_JOBID}.out
    

    还是不能计算

    Time = 0.005
    
    Courant Number mean: 0.000119441 max: 0.017568
    smoothSolver:  Solving for Ux, Initial residual = 1, Final residual = 2.06726e-09, No Iterat\
    ions 2
    smoothSolver:  Solving for Uy, Initial residual = 1, Final residual = 1.30712e-09, No Iterat\
    ions 2
    smoothSolver:  Solving for Uz, Initial residual = 1, Final residual = 1.31187e-09, No Iterat\
    ions 2
    GAMG:  Solving for p, Initial residual = 1, Final residual = 0.035832, No Iterations 36
    time step continuity errors : sum local = 8.55925e-06, global = -9.25811e-08, cumulative = -\
    9.25811e-08
    GAMG:  Solving for p, Initial residual = 0.000128631, Final residual = 6.05672e-06, No Itera\
    tions 39
    time step continuity errors : sum local = 4.27446e-07, global = -5.85326e-09, cumulative = -\
    9.84343e-08
    GAMG:  Solving for p, Initial residual = 8.87864e-06, Final residual = 9.28164e-08, No Itera\
    tions 74
    time step continuity errors : sum local = 6.57138e-09, global = 9.95505e-13, cumulative = -9\
    .84333e-08
    --------------------------------------------------------------------------
    mpiexec noticed that process rank 105 with PID 0 on node cp1056 exited on signal 8 (Floating\
     point exception).
    --------------------------------------------------------------------------
    

    还请有经验的朋友分析一下这个问题。



  • @random_ran

    Sorry, Chinese input missing.

    mpiexec noticed that process rank 105 with PID 0 on node cp1056 exited on signal 8 (Floating\
     point exception).
    

    This is a quite general error, which does not mean it comes from setting up of mpi. Meanwhile, sometimes your simulation blows up in parallel running, see: link text.

    Which version of OpenFOAM do you use? Could u please try PBiCG instead of smoothSolver? How do you decompose your grid?


  • 版主

    @cfd-china
    补充信息:

    OpenFOAM v4.1

    
    /*--------------------------------*- C++ -*----------------------------------*\                                                                                            
    | =========                 |                                                 |                                                                                            
    | \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |                                                                                            
    |  \\    /   O peration     | Version:  4.x                                   |                                                                                            
    |   \\  /    A nd           | Web:      www.OpenFOAM.org                      |                                                                                            
    |    \\/     M anipulation  |                                                 |                                                                                            
    \*---------------------------------------------------------------------------*/
    FoamFile
    {
        version     2.0;
        format      ascii;
        class       dictionary;
        location    "system";
        object      decomposeParDict;
    }
    // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //                                                                                            
    
    numberOfSubdomains 288;
    
    method          hierarchical;
    
    hierarchicalCoeffs
    {
        n               (16 9 2);
        delta           0.001;
        order           xyz;
    
    
    hierarchicalCoeffs
    {
        n               (16 9 2);
        delta           0.001;
        order           xyz;
    }
    
    manualCoeffs
    {
        dataFile        "cellDecomposition";
    }
    
    
    // ************************************************************************* //       
    
    
    *--------------------------------*- C++ -*----------------------------------*\                                                                                            
    | =========                 |                                                 |                                                                                            
    | \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |                                                                                            
    |  \\    /   O peration     | Version:  4.x                                   |                                                                                            
    |   \\  /    A nd           | Web:      www.OpenFOAM.org                      |                                                                                            
    |    \\/     M anipulation  |                                                 |                                                                                            
    \*---------------------------------------------------------------------------*/
    FoamFile
    {
        version     2.0;
        format      ascii;
        class       dictionary;
        location    "system";
        object      fvSolution;
    }
    // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //                                                                                            
    
    solvers
    {
        p
        {
            solver          GAMG;
            tolerance       1e-07;
            relTol          0.05;
            smoother        GaussSeidel;
        }
    
        pFinal
        {
            $p;
            smoother        DICGaussSeidel;
            tolerance       1e-07;
    	relTol          0;
        }
    
        "(U|k|B|nuTilda|s)"
        {
            solver          smoothSolver;
            smoother        GaussSeidel;
            tolerance       1e-07;
            relTol          0;
        }
    }
    
    PISO
    {
        nCorrectors     3;
        nNonOrthogonalCorrectors 0;
    }
    
    relaxationFactors
    {
      "U.*"               1;
      "nuTilda.*"         1;
    }
    
    

  • 版主

    再补充:

    192 case 的雷诺数在10^4
    288 case 的在10^5

    这两个case的分解方法,controDict, fvSchemes, fvSolution 基本一样。
    不同的是288 case 的时间步长更小而已。


  • 版主

    k 文件 tubulent kinetic energy:

    *--------------------------------*- C++ -*----------------------------------*\                  
    | =========                 |                                                 |                  
    | \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |                  
    |  \\    /   O peration     | Version:  4.x                                   |                  
    |   \\  /    A nd           | Web:      www.OpenFOAM.org                      |                  
    |    \\/     M anipulation  |                                                 |                  
    \*---------------------------------------------------------------------------*/
    FoamFile
    {
        version     2.0;
        format      ascii;
        class       volScalarField;
        object      k;
    }
    // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //                  
    
    dimensions      [0 2 -2 0 0 0 0];
    
    internalField   uniform 0;
    
    boundaryField
    {
        INLET
        {
            type            fixedValue;
            value           uniform 2e-05;
        }
    
        OUTLET
        {
            type            inletOutlet;
            inletValue      uniform 0;
            value           uniform 0;
        }
    
    
        BACK_CYC
        {
            type            cyclic;
        }
        FRONT_CYC
        {
            type            cyclic;
        }
    
        CYLINDER
        {
            type            fixedValue;
            value           uniform 0;
        }
    
    }
    

  • 版主

    @cfd-china

    非常感谢你的建议!非常管用,起码现在能跑起来了。

    
    Time = 0.005
    
    Courant Number mean: 0.000119441 max: 0.017568
    DILUPBiCG:  Solving for Ux, Initial residual = 1, Final residual = 2.34151e-11, No Iterations 2
    DILUPBiCG:  Solving for Uy, Initial residual = 1, Final residual = 1.61665e-10, No Iterations 2
    DILUPBiCG:  Solving for Uz, Initial residual = 1, Final residual = 1.63882e-10, No Iterations 2
    GAMG:  Solving for p, Initial residual = 1, Final residual = 0.035832, No Iterations 36
    time step continuity errors : sum local = 8.55925e-06, global = -9.25811e-08, cumulative = -9.25811e-08
    GAMG:  Solving for p, Initial residual = 0.000128631, Final residual = 6.05672e-06, No Iterations 39
    time step continuity errors : sum local = 4.27446e-07, global = -5.85326e-09, cumulative = -9.84343e-08
    GAMG:  Solving for p, Initial residual = 8.87864e-06, Final residual = 9.28164e-08, No Iterations 74
    time step continuity errors : sum local = 6.57138e-09, global = 9.95505e-13, cumulative = -9.84333e-08
    DILUPBiCG:  Solving for k, Initial residual = 0.297752, Final residual = 7.56559e-07, No Iterations 4
    bounding k, min: -2.24189e-08 max: 2e-05 average: 2.54114e-09
    ExecutionTime = 42.97 s  ClockTime = 43 s
    

    我改动的fvSolution 部分是:

        "(U|k|B|nuTilda|s)"
        {
            solver          PBiCG;
            preconditioner  DILU
            tolerance       1e-07;
            relTol          0;
        }
    

    目前运行了半个小时了。

    我知道后面还会有的新的问题。


登录后回复
 

与 CFD中文网 的连接断开,我们正在尝试重连,请耐心等待