With your answers being simply yes or no, and the size of your dataset, the whole process can be performed entirely within memory and in a constant time per question.

For your dataset 20,000 x 3,000, that requires 60 MB of ram and will determine the answer to any question in an average of about 250 milliseconds.

If both the number of respondents and the number of questions doubled, 40,000 x 6,000, this would rise to just 240 MB and 1/2 a second.

In terms of scalability, a 2GB memory machine should be able to handle 200,000 respondants x 10,000 questions, or well over half a million respondants at 3,000 questions apiece and process each question (on my 2GHz machine) at the rate of 250/millisecond or quarter of a million responses per second! (See 402390 for my excuses for this mistake.)

Please note this is for any question (within the limited framework I have code for), no matter how complex. Eg.

• 01. How many respondents answered question 0
• 02. How many respondents did not answer question 0
• 03. How many respondents answered question 10
• 04. How many respondents answered question 100
• 06. How many respondents answered 1000, 2000 but not 2500?
• 07. How many respondents answered every 10th question?
• 08. How many respondants answered question 3000?
• 09. How many respondants answered 123, 456, or 789 but not 135, 246, 357 or 468?
• 10. How many respondants answered all of 1000, 1010, 1020 but none of 2000, 2010, 2020?

To do this, I encode each respondants answers positionally in a string. If they answered yes to question 65, set the 64th byte of a 3,000 character string of 0's to 1. The same for all the others. The response strings are stored in a simple array. 20,000 strings of 3,000 bytes requires almost exactly 60 MB.

The question is then encoded in a similar way.

I've coded for questions of the form

any( these ) and not any( those )
all( these ) and not any( those )
any( these ) and not all( those )
all( these ) and not all( those )
[download]

Where 'these' and 'those' can be a list of any number of quesion numbers.

An empty all( ) for the 'these' part of the question effectively passes all and the count becomes entirely dependant upon the second (negative) part of the question, as with question 2.

An empty any( ) for the second part makes the question entirely dependant on the first part as with Qs 1,3,4, 8.

This could easily be extended, but is surprisingly flexible as is, allowing all the above questions to be answered.

Uncommenting the line inside the loop allow you to obtain the id's of the respondants matching the query, in addition to the counts. Building the array slows it down slightly.

The code:

#! perl -slw
use strict;
use Benchmark::Timer;
my $T = new Benchmark::Timer;

$|=1;

our $RESPONDENTS ||= 20_000;
our $QUESTIONS   ||=  3_000;

sub rndStr{ join'', @_[ map{ rand @_ } 1 .. shift ] }
sub makeMask { 
    my $compiled = '0' x $QUESTIONS;
    substr $compiled, $_, 1, '1' for @_;
    return $compiled;
}

sub any { ## ( set, mask ) = @_;
    return ( $_[ 0 ] & $_[ 1 ] ) != 0 ? 1 : 0;
}

sub all { ## ( set, mask ) = @_;
    return ( $_[ 0 ] & $_[ 1 ] ) eq $_[ 1 ] ? 1 : 0;
}

my %func; @func{ 'any', 'all' } = ( \&any, \&all );

my @data;
push @data, rndStr( $QUESTIONS, (0)x1, 1 ) for 1 .. $RESPONDENTS;

my $re_parse = qr[^ \s* ( (?ix-sm: any | all ) ) \( \s* ( [^)]+ ) \s* 
+\) \s* $]x;
my( %counts, %repondants );
while( my $question = <DATA> ) {
    my( $iMethod, $iSet ) = <DATA> =~ $re_parse;
    my $inclusive = $1 ? makeMask( split ' ', $iSet ) : undef;
    my( $xMethod, $xSet ) = <DATA> =~ $re_parse;
    my $exclusive = $1 ? makeMask( split ' ', $xSet ) : undef;

    $T->start( $question );

            $func{ $iMethod }->( $_, $inclusive )
    and not $func{ $xMethod }->( $_, $exclusive )
    and $counts{ $question }++
#    and push @{ $respondants{ $question } }, $_
    for @data;

    $T->stop( $question );
}

$T->report;
print "$_: $counts{ $_ }" for sort keys %counts;

__DATA__
01. How many respodents answered question 0
all( 0 )
any( )
02. How many respondents did not answer question 0
all( )
all(0)
03. How many respodents answered question 10
all( 10 )
any(  )
04. How many respodents answered question 100
all( 100 )
any(  )
04. How many respondant answered questions 14, 57, 103?
all( 14 57 103 )
all( )
06. How many respondent answered 1000, 2000 but not 2500?
all( 1000 2000 )
all( 2500 )
07. How many respondents answered every 10th question?
all( 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 19
+0 200 210 220 230 240 250 260 270 280 290 300 310 320 330 340 350 360
+ 370 380 390 400 410 420 430 440 450 460 470 480 490 500 510 520 530 
+540 550 560 570 580 590 600 610 620 630 640 650 660 670 680 690 700 7
+10 720 730 740 750 760 770 780 790 800 810 820 830 840 850 860 870 88
+0 890 900 910 920 930 940 950 960 970 980 990 1000 1010 1020 1030 104
+0 1050 1060 1070 1080 1090 1100 1110 1120 1130 1140 1150 1160 1170 11
+80 1190 1200 1210 1220 1230 1240 1250 1260 1270 1280 1290 1300 1310 1
+320 1330 1340 1350 1360 1370 1380 1390 1400 1410 1420 1430 1440 1450 
+1460 1470 1480 1490 1500 1510 1520 1530 1540 1550 1560 1570 1580 1590
+ 1600 1610 1620 1630 1640 1650 1660 1670 1680 1690 1700 1710 1720 173
+0 1740 1750 1760 1770 1780 1790 1800 1810 1820 1830 1840 1850 1860 18
+70 1880 1890 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 2
+010 2020 2030 2040 2050 2060 2070 2080 2090 2100 2110 2120 2130 2140 
+2150 2160 2170 2180 2190 2200 2210 2220 2230 2240 2250 2260 2270 2280
+ 2290 2300 2310 2320 2330 2340 2350 2360 2370 2380 2390 2400 2410 242
+0 2430 2440 2450 2460 2470 2480 2490 2500 2510 2520 2530 2540 2550 25
+60 2570 2580 2590 2600 2610 2620 2630 2640 2650 2660 2670 2680 2690 2
+700 2710 2720 2730 2740 2750 2760 2770 2780 2790 2800 2810 2820 2830 
+2840 2850 2860 2870 2880 2890 2900 2910 2920 2930 2940 2950 2960 2970
+ 2980 2990 3000 )
any( )
08. How many respondants answered question 3000?
all( 2999 )
any( )
09. How many respondants answered 123, 456, or 789 but not 135, 246, 3
+57 or 468?
any( 123 456 789 )
any( 135 246 468 )
10. How many respondants answered all of 1000, 1010, 1020 but none of 
+2000, 2010, 2020?
all( 1000 1010 1020 ) 
any( 2000 2010 2020 )
[download]

The results applied to a simulated dataset match the size desccribed:

P:\test>401728
  1 trial  of 01. How many respodents answered question 0
 (       234.375ms total),        234.375ms/trial

  1 trial  of 02. How many respondents did not answer question 0
 (       187.500ms total),        187.500ms/trial

  1 trial  of 03. How many respodents answered question 10
 (           250ms total),            250ms/trial

  1 trial  of 04. How many respodents answered question 100
 (       234.375ms total),        234.375ms/trial

  1 trial  of 04. How many respondant answered questions 14, 57, 103?
 (        93.750ms total),         93.750ms/trial

  1 trial  of 06. How many respondent answered 1000, 2000 but not 2500
+?
 (           125ms total),            125ms/trial

  1 trial  of 07. How many respondents answered every 10th question?
 (        62.500ms total),         62.500ms/trial

  1 trial  of 08. How many respondants answered question 3000?
 (           250ms total),            250ms/trial

  1 trial  of 09. How many respondants answered 123, 456, or 789 but n
+ot 135, 246, 357 or 468?
 (       703.125ms total),        703.125ms/trial

  1 trial  of 10. How many respondants answered all of 1000, 1010, 102
+0 but none of 2000, 2010, 2020?
 (       156.250ms total),        156.250ms/trial

01. How many respodents answered question 0
: 10067
02. How many respondents did not answer question 0
: 9933
03. How many respodents answered question 10
: 9797
04. How many respodents answered question 100
: 10068
06. How many respondent answered 1000, 2000 but not 2500?
: 2561
08. How many respondants answered question 3000?
: 9969
09. How many respondants answered 123, 456, or 789 but not 135, 246, 3
+57 or 468?
: 1121
10. How many respondants answered all of 1000, 1010, 1020 but none of 
+2000, 2010, 2020?
: 293
[download]

Note: Question 7 produces no results as no records matched.

The entire dataset can be stored pre-encoded in a single 60 MB file. in the event that your dataset grew beyond the capacity of memory (approx. 3/4 of a million respondants with 3000 answers on a machine with 2GB of ram), then simple switching to using Tie::File and processing the data in a single pass from disk would still easily out perform an (R|SQL)DB(MS).