http://qs321.pair.com?node_id=655655


in reply to Re^5: Reading huge file content
in thread Reading huge file content

The Inline C code below queries and displays the global memory stats (in 4096 byte pages) on my system.

It then allocates and frees successively larger chunks of virtual ram until the attempt fails.

The output looks like this: 

c:\test>valloc
Global Memory Status
Total Physical:392956 pages     Available Physical:309695 pages   
    ## 1.5 GB physical with ~1.3 GB free
Total PageFile:1048575 pages    Available PageFile:1048575 pages  
    ## 4 GB of pagefile all free
Total Virtual :524256 pages     Available Virtual :516124 pages   
    ## 2 GB of Virtual Memory Available mostly free

Allocated: 260000 pages [1064960000 bytes] at 280d0000
Allocated: 270000 pages [1105920000 bytes] at 280d0000
Allocated: 280000 pages [1146880000 bytes] at 280d0000
Allocated: 290000 pages [1187840000 bytes] at 280d0000
Allocated: 300000 pages [1228800000 bytes] at 280d0000
Allocated: 310000 pages [1269760000 bytes] at 280d0000
Allocated: 320000 pages [1310720000 bytes] at 280d0000
Not enough storage is available to process this command 
       at c:\test\valloc.pl line 18, <STDIN> line 7.

[download]

Note the close correspondance between the 320,000 pages successfully allocated and the 309,695/392,956 physical memory available. This is not a coincidence. (YMMV on other OSs). 

#! perl -slw
use strict;
#use Inline 'FORCE';
use Inline C => 'DATA', NAME => 'valloc', CLEAN_AFTER_BUILD => 0;
use constant PAGE => 4096;

my( $tPhys, $aPhys, $tPage, $aPage, $tVirt, $aVirt )
    = globalMemoryStatus();
print <<FMT;
Global Memory Status
Total Physical:$tPhys pages\tAvailable Physical:$aPhys pages
Total PageFile:$tPage pages\tAvailable PageFile:$aPage pages
Total Virtual :$tVirt pages\tAvailable Virtual :$aVirt pages
FMT

for my $nPages ( map $_ * 10000, 26 .. 52 ) {
    my $alloc = $nPages * PAGE;
    my $addr = virtualAlloc( $alloc ) or die $^E;
    printf "Allocated: $nPages pages [$alloc bytes] at %x", $addr; <ST
+DIN>;
    virtualFree( $addr ) or die $^E;
}

__DATA__
__C__
#include <windows.h>

void globalMemoryStatus ( ) {
    Inline_Stack_Vars;
    MEMORYSTATUS stat;
    GlobalMemoryStatus( &stat );
    Inline_Stack_Reset;
    Inline_Stack_Push( sv_2mortal( newSVuv( stat.dwTotalPhys / 4096 ) 
+) );
    Inline_Stack_Push( sv_2mortal( newSVuv( stat.dwAvailPhys / 4096 ) 
+) );
    Inline_Stack_Push( sv_2mortal( newSVuv( stat.dwTotalPageFile / 409
+6 ) ) );
    Inline_Stack_Push( sv_2mortal( newSVuv( stat.dwAvailPageFile / 409
+6 ) ) );
    Inline_Stack_Push( sv_2mortal( newSVuv( stat.dwTotalVirtual / 4096
+ ) ) );
    Inline_Stack_Push( sv_2mortal( newSVuv( stat.dwAvailVirtual / 4096
+ ) ) );
    Inline_Stack_Done;
    return;
}

U32 virtualAlloc( U32 size ) {
    return (U32)VirtualAlloc( NULL, (SIZE_T)size,  
        MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE 
    );
}

U32 virtualFree( U32 a ) {
    return VirtualFree( (LPVOID)a, 0, MEM_RELEASE );
}

[download]
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^7: Reading huge file content
by GrandFather (Saint) on Dec 09, 2007 at 07:54 UTC

    I couldn't get the inline C version of your code to work on my system so I rewrote the whole thing in C. I then ran the program twice, increasing the page file size by 1 GB between runs with the following result:

    Global Memory Status:
Total Physical: 401981440 bytes Available Physical: 32329728 bytes
Total PageFile: 1569083392 bytes        Available PageFile: 575242240 
+bytes
Total Virtual : 2147352576 bytes        Available Virtual : 2139758592
+ bytes
Allocated: 16000 pages [65536000 bytes] at 0x00420000
Allocated: 26000 pages [106496000 bytes] at 0x00420000
Allocated: 36000 pages [147456000 bytes] at 0x00420000
Allocated: 46000 pages [188416000 bytes] at 0x00420000
Allocated: 56000 pages [229376000 bytes] at 0x00420000
Allocated: 66000 pages [270336000 bytes] at 0x10320000
Allocated: 76000 pages [311296000 bytes] at 0x10320000
Allocated: 86000 pages [352256000 bytes] at 0x10320000
Allocated: 96000 pages [393216000 bytes] at 0x10320000
Allocated: 106000 pages [434176000 bytes] at 0x10320000
Allocated: 116000 pages [475136000 bytes] at 0x10320000
Allocated: 126000 pages [516096000 bytes] at 0x10320000
Allocated: 136000 pages [557056000 bytes] at 0x10320000
Failed to allocate 146000 pages [598016000 bytes]

C:\Documents and Settings\Peter\My Documents\Visual Studio 2005\Projec
+ts\vmtest\vmtest\Debug>v
Global Memory Status:
Total Physical: 401981440 bytes Available Physical: 21180416 bytes
Total PageFile: -1811525632 bytes       Available PageFile: 1483104256
+ bytes
Total Virtual : 2147352576 bytes        Available Virtual : 2139758592
+ bytes
Allocated: 16000 pages [65536000 bytes] at 0x00420000
Allocated: 26000 pages [106496000 bytes] at 0x00420000
Allocated: 36000 pages [147456000 bytes] at 0x00420000
Allocated: 46000 pages [188416000 bytes] at 0x00420000
Allocated: 56000 pages [229376000 bytes] at 0x00420000
Allocated: 66000 pages [270336000 bytes] at 0x10320000
Allocated: 76000 pages [311296000 bytes] at 0x10320000
Allocated: 86000 pages [352256000 bytes] at 0x10320000
Allocated: 96000 pages [393216000 bytes] at 0x10320000
Allocated: 106000 pages [434176000 bytes] at 0x10320000
Allocated: 116000 pages [475136000 bytes] at 0x10320000
Allocated: 126000 pages [516096000 bytes] at 0x10320000
Allocated: 136000 pages [557056000 bytes] at 0x10320000
Allocated: 146000 pages [598016000 bytes] at 0x10320000
Allocated: 156000 pages [638976000 bytes] at 0x10320000
Allocated: 166000 pages [679936000 bytes] at 0x10320000
Allocated: 176000 pages [720896000 bytes] at 0x10320000
Allocated: 186000 pages [761856000 bytes] at 0x10320000
Allocated: 196000 pages [802816000 bytes] at 0x10320000
Allocated: 206000 pages [843776000 bytes] at 0x10320000
Allocated: 216000 pages [884736000 bytes] at 0x10320000
Allocated: 226000 pages [925696000 bytes] at 0x10320000
Allocated: 236000 pages [966656000 bytes] at 0x10320000
Allocated: 246000 pages [1007616000 bytes] at 0x10320000
Allocated: 256000 pages [1048576000 bytes] at 0x10320000
Allocated: 266000 pages [1089536000 bytes] at 0x10320000
Allocated: 276000 pages [1130496000 bytes] at 0x10320000
Allocated: 286000 pages [1171456000 bytes] at 0x10320000
Allocated: 296000 pages [1212416000 bytes] at 0x10320000
Allocated: 306000 pages [1253376000 bytes] at 0x10320000
Allocated: 316000 pages [1294336000 bytes] at 0x10320000
Allocated: 326000 pages [1335296000 bytes] at 0x10320000
Allocated: 336000 pages [1376256000 bytes] at 0x10320000
Allocated: 346000 pages [1417216000 bytes] at 0x10320000
Allocated: 356000 pages [1458176000 bytes] at 0x10320000
Allocated: 366000 pages [1499136000 bytes] at 0x10320000
Allocated: 376000 pages [1540096000 bytes] at 0x10320000
Allocated: 386000 pages [1581056000 bytes] at 0x10320000
Failed to allocate 396000 pages [1622016000 bytes]

C:\Documents and Settings\Peter\My Documents\Visual Studio 2005\Projec
+ts\vmtest\vmtest\Debug>

    [download]

    Assuming the code is behaving correctly it seems to be allocating a larger block than my system has physical RAM - in fact about 4 times more!

    The C code:

    #include <windows.h>
#include <stdio.h>

#define PAGE 4096


void *virtualAlloc (long size) 
{
return VirtualAlloc (0, size, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE
+);
}


BOOL virtualFree (void *a) {
    return VirtualFree (a, 0, MEM_RELEASE);
}


void main ()
{
MEMORYSTATUS stat;

GlobalMemoryStatus (&stat);

printf (
    "Global Memory Status:\r\n"
    "Total Physical: %d bytes\tAvailable Physical: %d bytes\r\n"
    "Total PageFile: %d bytes\tAvailable PageFile: %d bytes\r\n"
    "Total Virtual : %d bytes\tAvailable Virtual : %d bytes\r\n",
    stat.dwTotalPhys, stat.dwAvailPhys,
    stat.dwTotalPageFile, stat.dwAvailPageFile,
    stat.dwTotalVirtual, stat.dwAvailVirtual
    );

for (long nPages = 16000; nPages <= 520000; nPages += 10000) 
    {
    long allocSize = nPages * PAGE;
    void *addr = virtualAlloc (allocSize);

    if (! addr)
        {
        printf ("Failed to allocate %d pages [%d bytes]\r\n", nPages, 
+allocSize);
        break;
        }

    printf ("Allocated: %d pages [%d bytes] at 0x%08x\r\n", nPages, al
+locSize, addr);
    virtualFree (addr);
    }
}

    [download]
    Perl is environmentally friendly - it saves trees
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom