Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re^3: System call doesn't work when there is a large amount of data in a hash

by bliako (Monsignor)
on Apr 29, 2020 at 13:41 UTC ( [id://11116209]=note: print w/replies, xml ) Need Help??


in reply to Re^2: System call doesn't work when there is a large amount of data in a hash
in thread System call doesn't work when there is a large amount of data in a hash

It is very unlikely that blastn takes a GB-size sequence as input from the command line! Most likely what the command expects from you, is to provide the name of the file which holds that huge data.

So, in all likelihood, you must 1) write your hash to a file if it is indeed in the memory of the Perl script (and not in a file already!) and then 2) make the "system call" and provide the filename to it, as part of the command arguments.

Make sure that if the expected output is huge, to instruct blastn to write its output to a file. Do not read it back from the output of the command (stdout)! Perhaps use the -o outfile option or simply redirect your command to a file, which is not an elegant solution if you are doing it via Perl's system command.

The above procedure is acceptable if you create/calculate/transform that hash in the Perl script. Just to make sure: if you just read the hash from file, do not change it in any way and then blastn on it (which implies writing it to a file, as I recommend above), then you are doing something wrong.

Since you have a lot of RAM available, it is worth investigating either storing it in a RAM-disk which you have to create it first, in fact all your data could go there, including temporary files. OR, use memory-mapped files, perhaps read on File::Map.

bliako

Replies are listed 'Best First'.
Re^4: System call doesn't work when there is a large amount of data in a hash
by Nicolasd (Acolyte) on Apr 29, 2020 at 13:58 UTC

    Hi, Thanks for the reply

    The blastn is not related to the hash,the hash contains data from sequencing reads, the files for the blastn are very small

    The hash is being used constantly, so that why it needs be a hash, other methods like databases or writing to files, is too slow

    And if change the blastn system call to system("echo Hello"), I get the same problem, so it's not related to the bastn

    That RAM disk is maybe to complicated for users of the tool, mostly biologists who I need to explain what a terminal is :)

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11116209]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others avoiding work at the Monastery: (5)
As of 2024-04-25 10:12 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found