OF memory requirement



  • 大家好,

    请问在并行运算的时候,每个core需要多少memory呢?
    应该是和 decompse之后每个core所负责的cells数目 以及 solver复杂程度有关,请问有什么相关的大致规律呢?

    昨天发现一个大概3千万cells的mesh,单核跑checkMesh时 提示run out of memory, 但是solver可以跑!是不是checkMesh对memory要求很大?

    非常感谢!


  • OpenFOAM讲师

    我自己的算例: 1.44M的Cell 峰值 RSS 3.8 GB, 还有一些小的 case。

    圆柱绕流 pisofoam;
    DDES;
    OF4.1,;
    cpu: AMD Opteron™ Processor 617
    OS: CentOS Linux release 7.5.1804 (Core)

    =======info begin
    I am in:
    /home/newuser/scratch/lengthEffect/workingFolder/case_z_100Layer
    JobID Elapsed NCPUS AveRSS MaxRSS AveVMSize ReqMem


    13699.batch 21:35:11 24 3743510K 3800206K 21826016K 31Gn
    Case mesh contains
    cells: 1440000

    I am in:
    /home/newuser/scratch/lengthEffect/workingFolder/case_z_20Layer
    JobID Elapsed NCPUS AveRSS MaxRSS AveVMSize ReqMem


    13700.batch 03:35:13 24 1913933K 1953179K 19062444K 31Gn
    Case mesh contains
    cells: 288000

    I am in:
    /home/newuser/scratch/lengthEffect/workingFolder/case_z_40Layer
    JobID Elapsed NCPUS AveRSS MaxRSS AveVMSize ReqMem


    13701.batch 09:46:27 24 2329258K 2405771K 19580912K 31Gn
    Case mesh contains
    cells: 576000

    I am in:
    /home/newuser/scratch/lengthEffect/workingFolder/case_z_60Layer
    JobID Elapsed NCPUS AveRSS MaxRSS AveVMSize ReqMem


    13702.batch 16:42:47 24 2777944K 2875499K 20423644K 31Gn
    Case mesh contains
    cells: 864000

    I am in:
    /home/newuser/scratch/lengthEffect/workingFolder/case_z_80Layer
    JobID Elapsed NCPUS AveRSS MaxRSS AveVMSize ReqMem


    13703.batch 18:27:01 24 2225055K 3324466K 15583600K 31Gn
    Case mesh contains
    cells: 1152000

    =======info end


  • OpenFOAM讲师

    又做了个测试,转换网格和checkMesh都没有问题,createPatch出现了如下错误:

    // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
    Create time
    
    Create polyMesh for time = 0
    
    Reading createPatchDict
    
    Adding new patch FRONT_CYC as patch 5 from 
    {
        type            cyclic;
        neighbourPatch  BACK_CYC;
        matchTolerance  0.01;
    }
    
    Adding new patch BACK_CYC as patch 6 from 
    {
        type            cyclic;
        neighbourPatch  FRONT_CYC;
        matchTolerance  0.01;
    }
    
    
    Moving faces from patch FRONT to patch 5
    terminate called after throwing an instance of 'std::bad_array_new_length'
      what():  std::bad_array_new_length
    

    我的c++不好,但还是想尝试去找这个error的源头。

    cppreference 上说三种可能的原因,我觉得我的case是“too large”。

    #include <iostream>
    #include <new>
    #include <climits>
     
    int main()
    {
        int negative = -1;
        int small = 1;
        int large = INT_MAX;
        try {
            new int[negative];           // negative size
            new int[small]{1,2,3};       // too many initializers
            new int[large][1000000];     // too large
        } catch(const std::bad_array_new_length &e) {
            std::cout << e.what() << '\n';
        }
    }
    

    我顺着 Moving faces from patch FRONT to patch 5 找到 createPatch.C 源代码:

    ====> createPatch.C:  Line 705~714
                    forAll(pp, i)
                    {
                        changePatchID
                        (
                            mesh,
                            pp.start() + i,
                            destPatchi,
                            meshMod
                        );
                    }
    

    似乎是死在了这个循环里,

    changePatchID 的arguments:
    
    (
        const polyMesh& mesh,
        const label faceID,
        const label patchID,
        polyTopoChange& meshMod
    )
    

    changePatchID 里嵌套的这个函数很可疑 meshMod.setAction, 特别是 polyModifyFace 这个函数

    定义在

    polyModifyFace.H 奇怪 polyModifyPoint.C 居然不存在。 似乎是一个虚函数。

    看到这里,感觉无能为力了,想要找到类似这种 new int[large][1000000]; // too large
    的声明看来是行不通了。

    所以想请教大家的想法。

    ====> checkMesh
    /*---------------------------------------------------------------------------*\
    | =========                 |                                                 |
    | \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
    |  \\    /   O peration     | Version:  4.1                                   |
    |   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
    |    \\/     M anipulation  |                                                 |
    \*---------------------------------------------------------------------------*/
    Build  : 4.1
    Exec   : checkMesh
    Date   : Jul 06 2018
    Time   : 15:54:53
    Host   : "cp0401"
    PID    : 12872
    Case   : /scratch/crazyuser/biggerMesh
    nProcs : 1
    sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
    fileModificationChecking : Monitoring run-time modified files using timeStampMaster
    allowSystemOperations : Allowing user-supplied system call operations
    
    // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
    Create time
    
    Create polyMesh for time = 0
    
    Time = 0
    
    Mesh stats
        points:           257514000
        faces:            769512000
        internal faces:   766488000
        cells:            256000000
        faces per cell:   6
        boundary patches: 5
        point zones:      0
        face zones:       1
        cell zones:       1
    
    Overall number of cells of each type:
        hexahedra:     256000000
        prisms:        0
        wedges:        0
        pyramids:      0
        tet wedges:    0
        tetrahedra:    0
        polyhedra:     0
    
    Checking topology...
        Boundary definition OK.
        Cell to face addressing OK.
        Point usage OK.
        Upper triangular ordering OK.
        Face vertices OK.
        Number of regions: 1 (OK).
    
    Checking patch topology for multiply connected surfaces...
        Patch               Faces    Points   Surface topology                  
        FRONT               1000000  1002000  ok (non-closed singly connected)  
        INLET               256000   257257   ok (non-closed singly connected)  
        OUTLET              256000   257257   ok (non-closed singly connected)  
        CYLINDER            512000   514000   ok (non-closed singly connected)  
        BACK                1000000  1002000  ok (non-closed singly connected)  
    
    Checking geometry...
        Overall domain bounding box (-32 -32 0) (32 32 3.33332)
        Mesh has 3 geometric (non-empty/wedge) directions (1 1 1)
        Mesh has 3 solution (non-empty) directions (1 1 1)
        Boundary openness (-4.65606e-16 4.16736e-21 -4.97036e-16) OK.
        Max cell openness = 3.18122e-16 OK.
        Max aspect ratio = 29.4282 OK.
        Minimum face area = 1.3175e-06. Maximum face area = 0.0385213.  Face area magnitudes OK.
        Min volume = 1.71549e-08. Max volume = 0.000501578.  Total volume = 10720.6.  Cell volumes OK.
        Mesh non-orthogonality Max: 1.72876e-05 average: 0
        Non-orthogonality check OK.
        Face pyramids OK.
        Max skewness = 0.00266764 OK.
        Coupled point location match (average 0) OK.
    
    Mesh OK.
    
    End
    

  • OpenFOAM讲师

    又查了一下峰值内存,似乎感觉到了问题的所在。

    JobID    Elapsed      NCPUS     AveRSS     MaxRSS  AveVMSize     ReqMem 
    ------------ ---------- ---------- ---------- ---------- ---------- ---------- 
    13898.batch    05:56:36         48 191885982K 194887914K 192155004K      200Gn 
    

  • OpenFOAM讲师

    重新运行了算例, 提高了内存请求(200GB->251GB)。同样的结果。请求内存似乎不是问题所在,峰值 RSS 和 200G内存一样。

           JobID    Elapsed      NCPUS     AveRSS     MaxRSS  AveVMSize     ReqMem 
    ------------ ---------- ---------- ---------- ---------- ---------- ---------- 
    14389          06:00:15        192                                       251Gn 
    14389.batch    06:00:15         48       302K 194452190K    113468K      251Gn 
    14389.extern   06:00:15        192       115K       144K    107948K      251Gn