Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re^3: Reading huge file content

by GrandFather (Saint)
on Dec 06, 2007 at 23:58 UTC ( #655554=note: print w/replies, xml ) Need Help??


in reply to Re^2: Reading huge file content
in thread Reading huge file content

Yes, you are wrong. ;)

Approximately: virtual memory systems give a process a chunk of memory that looks to the process like one large contiguous piece of memory. The process can not tell where or how its process space memory is mapped to physical memory. Indeed that mapping may change from access to access. The OS takes care of ensuring that when a process accesses a chunk of process space memory that the access succeeds if possible. That may entail writing a chunk of physical memory to disk, reading another chunk of disk to get the accessed process memory's contents, then fixing up tables to map process memory address to physical addresses. Hitting on the VM doesn't come cheap!


Perl is environmentally friendly - it saves trees

Replies are listed 'Best First'.
Re^4: Reading huge file content
by BrowserUk (Patriarch) on Dec 07, 2007 at 05:00 UTC

    I'd check again.

    Virtual memory will allow your processes to allocate more memory than you have physical ram, provided that you do not need to process it all as a single contiguous chunk.

    But the largest single contiguous chunk (eg. string) that you will be able to allocate will always be somewhat less than your total physical ram.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

        The Inline C code below queries and displays the global memory stats (in 4096 byte pages) on my system.

        It then allocates and frees successively larger chunks of virtual ram until the attempt fails.

        The output looks like this:

        c:\test>valloc Global Memory Status Total Physical:392956 pages Available Physical:309695 pages ## 1.5 GB physical with ~1.3 GB free Total PageFile:1048575 pages Available PageFile:1048575 pages ## 4 GB of pagefile all free Total Virtual :524256 pages Available Virtual :516124 pages ## 2 GB of Virtual Memory Available mostly free Allocated: 260000 pages [1064960000 bytes] at 280d0000 Allocated: 270000 pages [1105920000 bytes] at 280d0000 Allocated: 280000 pages [1146880000 bytes] at 280d0000 Allocated: 290000 pages [1187840000 bytes] at 280d0000 Allocated: 300000 pages [1228800000 bytes] at 280d0000 Allocated: 310000 pages [1269760000 bytes] at 280d0000 Allocated: 320000 pages [1310720000 bytes] at 280d0000 Not enough storage is available to process this command at c:\test\valloc.pl line 18, <STDIN> line 7.

        Note the close correspondance between the 320,000 pages successfully allocated and the 309,695/392,956 physical memory available. This is not a coincidence. (YMMV on other OSs).


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://655554]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (4)
As of 2022-01-29 08:42 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:












    Results (74 votes). Check out past polls.

    Notices?