OF memory requirement

sibo

大家好，

请问在并行运算的时候，每个core需要多少memory呢？
应该是和 decompse之后每个core所负责的cells数目以及 solver复杂程度有关，请问有什么相关的大致规律呢？

昨天发现一个大概3千万cells的mesh，单核跑checkMesh时提示run out of memory, 但是solver可以跑！是不是checkMesh对memory要求很大？

非常感谢！

random_ran

我自己的算例: 1.44M的Cell 峰值 RSS 3.8 GB, 还有一些小的 case。

圆柱绕流 pisofoam；
DDES；
OF4.1,；
cpu: AMD Opteron™ Processor 617
OS: CentOS Linux release 7.5.1804 (Core)

=======info begin
I am in:
/home/newuser/scratch/lengthEffect/workingFolder/case_z_100Layer
JobID Elapsed NCPUS AveRSS MaxRSS AveVMSize ReqMem

13699.batch 21:35:11 24 3743510K 3800206K 21826016K 31Gn
Case mesh contains
cells: 1440000

I am in:
/home/newuser/scratch/lengthEffect/workingFolder/case_z_20Layer
JobID Elapsed NCPUS AveRSS MaxRSS AveVMSize ReqMem

13700.batch 03:35:13 24 1913933K 1953179K 19062444K 31Gn
Case mesh contains
cells: 288000

I am in:
/home/newuser/scratch/lengthEffect/workingFolder/case_z_40Layer
JobID Elapsed NCPUS AveRSS MaxRSS AveVMSize ReqMem

13701.batch 09:46:27 24 2329258K 2405771K 19580912K 31Gn
Case mesh contains
cells: 576000

I am in:
/home/newuser/scratch/lengthEffect/workingFolder/case_z_60Layer
JobID Elapsed NCPUS AveRSS MaxRSS AveVMSize ReqMem

13702.batch 16:42:47 24 2777944K 2875499K 20423644K 31Gn
Case mesh contains
cells: 864000

I am in:
/home/newuser/scratch/lengthEffect/workingFolder/case_z_80Layer
JobID Elapsed NCPUS AveRSS MaxRSS AveVMSize ReqMem

13703.batch 18:27:01 24 2225055K 3324466K 15583600K 31Gn
Case mesh contains
cells: 1152000

=======info end

random_ran

又做了个测试，转换网格和checkMesh都没有问题，createPatch出现了如下错误:

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create polyMesh for time = 0

Reading createPatchDict

Adding new patch FRONT_CYC as patch 5 from 
{
    type            cyclic;
    neighbourPatch  BACK_CYC;
    matchTolerance  0.01;
}

Adding new patch BACK_CYC as patch 6 from 
{
    type            cyclic;
    neighbourPatch  FRONT_CYC;
    matchTolerance  0.01;
}


Moving faces from patch FRONT to patch 5
terminate called after throwing an instance of 'std::bad_array_new_length'
  what():  std::bad_array_new_length

我的c++不好，但还是想尝试去找这个error的源头。

cppreference 上说三种可能的原因，我觉得我的case是“too large”。

#include <iostream>
#include <new>
#include <climits>
 
int main()
{
    int negative = -1;
    int small = 1;
    int large = INT_MAX;
    try {
        new int[negative];           // negative size
        new int[small]{1,2,3};       // too many initializers
        new int[large][1000000];     // too large
    } catch(const std::bad_array_new_length &e) {
        std::cout << e.what() << '\n';
    }
}

我顺着 Moving faces from patch FRONT to patch 5 找到 createPatch.C 源代码:

====> createPatch.C:  Line 705~714
                forAll(pp, i)
                {
                    changePatchID
                    (
                        mesh,
                        pp.start() + i,
                        destPatchi,
                        meshMod
                    );
                }

似乎是死在了这个循环里，

changePatchID 的arguments：

(
    const polyMesh& mesh,
    const label faceID,
    const label patchID,
    polyTopoChange& meshMod
)

changePatchID 里嵌套的这个函数很可疑 meshMod.setAction，特别是 polyModifyFace 这个函数

定义在

polyModifyFace.H 奇怪 polyModifyPoint.C 居然不存在。似乎是一个虚函数。

看到这里，感觉无能为力了，想要找到类似这种 new int[large][1000000]; // too large
的声明看来是行不通了。

所以想请教大家的想法。

====> checkMesh
/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  4.1                                   |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 4.1
Exec   : checkMesh
Date   : Jul 06 2018
Time   : 15:54:53
Host   : "cp0401"
PID    : 12872
Case   : /scratch/crazyuser/biggerMesh
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create polyMesh for time = 0

Time = 0

Mesh stats
    points:           257514000
    faces:            769512000
    internal faces:   766488000
    cells:            256000000
    faces per cell:   6
    boundary patches: 5
    point zones:      0
    face zones:       1
    cell zones:       1

Overall number of cells of each type:
    hexahedra:     256000000
    prisms:        0
    wedges:        0
    pyramids:      0
    tet wedges:    0
    tetrahedra:    0
    polyhedra:     0

Checking topology...
    Boundary definition OK.
    Cell to face addressing OK.
    Point usage OK.
    Upper triangular ordering OK.
    Face vertices OK.
    Number of regions: 1 (OK).

Checking patch topology for multiply connected surfaces...
    Patch               Faces    Points   Surface topology                  
    FRONT               1000000  1002000  ok (non-closed singly connected)  
    INLET               256000   257257   ok (non-closed singly connected)  
    OUTLET              256000   257257   ok (non-closed singly connected)  
    CYLINDER            512000   514000   ok (non-closed singly connected)  
    BACK                1000000  1002000  ok (non-closed singly connected)  

Checking geometry...
    Overall domain bounding box (-32 -32 0) (32 32 3.33332)
    Mesh has 3 geometric (non-empty/wedge) directions (1 1 1)
    Mesh has 3 solution (non-empty) directions (1 1 1)
    Boundary openness (-4.65606e-16 4.16736e-21 -4.97036e-16) OK.
    Max cell openness = 3.18122e-16 OK.
    Max aspect ratio = 29.4282 OK.
    Minimum face area = 1.3175e-06. Maximum face area = 0.0385213.  Face area magnitudes OK.
    Min volume = 1.71549e-08. Max volume = 0.000501578.  Total volume = 10720.6.  Cell volumes OK.
    Mesh non-orthogonality Max: 1.72876e-05 average: 0
    Non-orthogonality check OK.
    Face pyramids OK.
    Max skewness = 0.00266764 OK.
    Coupled point location match (average 0) OK.

Mesh OK.

End

random_ran

又查了一下峰值内存，似乎感觉到了问题的所在。

JobID    Elapsed      NCPUS     AveRSS     MaxRSS  AveVMSize     ReqMem 
------------ ---------- ---------- ---------- ---------- ---------- ---------- 
13898.batch    05:56:36         48 191885982K 194887914K 192155004K      200Gn

random_ran

重新运行了算例, 提高了内存请求(200GB->251GB)。同样的结果。请求内存似乎不是问题所在，峰值 RSS 和 200G内存一样。

       JobID    Elapsed      NCPUS     AveRSS     MaxRSS  AveVMSize     ReqMem 
------------ ---------- ---------- ---------- ---------- ---------- ---------- 
14389          06:00:15        192                                       251Gn 
14389.batch    06:00:15         48       302K 194452190K    113468K      251Gn 
14389.extern   06:00:15        192       115K       144K    107948K      251Gn

OF memory requirement

13699.batch 21:35:11 24 3743510K 3800206K 21826016K 31Gn Case mesh contains cells: 1440000

13700.batch 03:35:13 24 1913933K 1953179K 19062444K 31Gn Case mesh contains cells: 288000

13701.batch 09:46:27 24 2329258K 2405771K 19580912K 31Gn Case mesh contains cells: 576000

13702.batch 16:42:47 24 2777944K 2875499K 20423644K 31Gn Case mesh contains cells: 864000

13703.batch 18:27:01 24 2225055K 3324466K 15583600K 31Gn Case mesh contains cells: 1152000

13699.batch 21:35:11 24 3743510K 3800206K 21826016K 31Gn
Case mesh contains
cells: 1440000

13700.batch 03:35:13 24 1913933K 1953179K 19062444K 31Gn
Case mesh contains
cells: 288000

13701.batch 09:46:27 24 2329258K 2405771K 19580912K 31Gn
Case mesh contains
cells: 576000

13702.batch 16:42:47 24 2777944K 2875499K 20423644K 31Gn
Case mesh contains
cells: 864000

13703.batch 18:27:01 24 2225055K 3324466K 15583600K 31Gn
Case mesh contains
cells: 1152000