Re: Private Utilities

Perfect! I have been looking for just such a solution this morn and appreciate your post - here is one I just put together really quick ... to chunk up a given fasta file (unaligned sequences) in to N files for submission of multiple smaller jobs on a cluster:

#!/usr/bin/perl -w
use strict;
my $num_ch  = $ARGV[0] or die "must pass the # of chunks $!";
open (FH,"<$ARGV[1]")  or die "must pass the file name to make chunks 
+of $!";
my @lines = ();
$/ = ">";
while (<FH>){
        chomp;
        next if /^$/; # get rid of blank lines
        push @lines,">$_";
}
close FH;
my $num_rec = scalar(@lines);
print "number of records : $num_rec\n";
#die "Chunks exceed records!!" unless ($num_rec >= $num_ch);
sub write_em {
        my $output = shift()."_chunk.fa";
        my $ar_ref = shift;
        open (QRTR,">$output") or die "Cannot open $output : $!";
        print QRTR @$ar_ref;
        close (QRTR) or die "cannot close $output : $!";
}
my $cnt = 0;
my $rng = int($num_rec/$num_ch) + ($num_rec%$num_ch ? 1 : 0);
for (1..$num_ch) {
        write_em($_,[@lines[$cnt..($_ == $num_ch ? $#lines : ($cnt+($r
+ng-1)))]]);
        $cnt += $rng;
}
[download]

Feel free to comment or point me in the right direction where I have gone astray - this just scratched a specific itch ...

Comment on Re: Private Utilities Download Code

Replies are listed 'Best First'.
Re^2: Private Utilities by thor (Priest) on Dec 01, 2005 at 20:06 UTC
From what I understand, these files can be large. I also see that you're essentially reading the whole file into memory. The combination of those two could be bad. Before I propose a better solution, is it okay to put adjacent lines in different files? That is, let's say that I have a file that contains the following: `AAA BBB CCC DDD EEE FFF` [download] Is it okay to split into two files like so: `AAA CCC EEE --- BBB DDD FFF` [download] If so, I may have a really slick solution floating in my head... thor `The only easy day was yesterday`	[reply] [d/l] [select]
Re^3: Private Utilities by l3v3l (Monk) on Dec 01, 2005 at 22:36 UTC
no problem!! round robin is fine - here is a way I have done it in a liner without having to read the entire file into mem (just an example) : `# Choose first N FASTA records perl -ne 'BEGIN {$/=">";$o=0}{chomp;$o<N?(/^$/?next:print">$_"):last;$ +o++}' EXAMPLE.fa` [download] example usage: `perl -ne 'BEGIN{$/=">";$o=0}{chomp;$o<22?(/^$/?next:print">$_"):last; +$o++}' b4x_est100.fa # print the first 22 records` [download]	[reply] [d/l] [select]
Re^4: Private Utilities by thor (Priest) on Dec 02, 2005 at 04:57 UTC
Alright...here's a simple multiplexer: `use strict; use warnings; use Getopt::Long; my ($number, $format) = (2, "output"); GetOptions( "number=i" => \$number, "format=s" => \$format, ); my @output_file; foreach my $num (1..$number) { my $file = "$format.$num"; open( $output_file[$num-1], ">", $file ) or die "Couldn't open + '$file' for write: $!\n"; } my $file_num = 0; while(<>) { print {$output_file[$file_num]} $_; $file_num += 1; $file_num %= $number; }` [download] Save it in a file of your choosing, and call it like: `perl script.pl --format out --number 7 input1 input2 ...` [download] thor `The only easy day was yesterday`	[reply] [d/l] [select]
Re^5: Private Utilities by l3v3l (Monk) on Dec 02, 2005 at 16:33 UTC
Re^6: Private Utilities by thor (Priest) on Dec 02, 2005 at 20:22 UTC


There's more than one way to do things
	PerlMonks