Multiplexing HTTPS server, peer cert authentication problem.

erroneousBollock has asked for the wisdom of the Perl Monks concerning the following question:

Dear esteemed monks,

I'm having some problems making SSL peer certificate authentication work in a multiplexing server -- my existing server mostly works (including peer certificate authentication), but...

The server's primary deployment target is (unfortunately) Windows. Request "handlers" can take between .05-20 seconds to do their work, so threads are needed to service incoming requests concurrently.

IO::Socket::SSL seems to have a lot of problems when its instances are used where the 'threads' module is imported. I have worked around this by writing the HTTPS server as a multiplexing (non-blocking) server, which is passed an object which allows the server to enqueue "work" to be done for the requests in other threads via Thread::Queue::Any (all this works without having threads imported in the multiplexing server's package).

My current server implementation (a subclass of HTTP::Daemon::SSL) is only non-blocking with respect to the socket->accept() method, so when multiple simultaneous requests (eg: images referenced in an HTML page) come in from a browser, some connections are never serviced (eg: they come in during a read/write).

To fix the problem, I've prototyped an entirely multiplexing server using IO::Select and IO::Socket::SSL (blocking). It's fast, robust (in my stress-testing) and can slot nicely into my existing server architecture.

Unfortunately, now that all operations don't block and that multiple requests are now being serviced simultaneously, the SSL peer certificate authentication functionality of IO::Socket::SSL (really of Net::SSLeay) no longer works.

Below are:

The demo server
A patch to IO::Socket::SSL for peer certificate authentication.
A set of keys and certificates for testing (or use your own)

After patching IO::Socket::SSL, put certs/ in the same directory as server.pl. You should import my-ca.pem, server-cert.pem and client-cert.pem into your browser -- this may require that you convert the certificate to p12 format. All certificates have the passphrase 'test'.

Any suggestions (including other approaches) are welcome... I've intentionally omitted a bunch of business constraints from the background so that I don't prematurely dismiss any good ideas.

Thanks,
-David.

server.pl

#!/usr/bin/perl

# <--- this is line 3
# Mutliplexing HTTP/S server to show IO::Socket::SSL problem.
#   for HTTP (working):
#     Comment lines 16,25,29,30,31,32
#     Uncomment lines 15,24
#   for HTTPS without peer cert authentication (working): 
#     Comment lines 15,24,30,32
#     Uncomment lines 16,25,29,31
#   for HTTPS with peer cert authentication (broken):
#     Comment lines 15,24,31
#     Uncomment lines 16,25,29,30,32

use IO::Socket::INET;
#use IO::Socket::SSL qw/debug4/;
use IO::Select;
use HTTP::Response;
use File::MMagic;
use strict;
use warnings;

my $magic = File::MMagic->new;
my $listen = IO::Socket::INET->new(
#my $listen = IO::Socket::SSL->new(
    LocalPort       => 2222,
    Listen          => 10,
    Reuse           => 1,
#    SSL_use_cert     => 1,
#    SSL_verify_depth => 1,    # verify CA->server, server->client
#    SSL_verify_mode  => 0x00, # works  (no peer cert auth)
#    SSL_verify_mode  => 0x03  # broken (peer cert auth)
  );

my $timeout = 0.002;    
my $rlen    = 1024;
my $wlen    = 4096;
my $fblen   = 10240;

my $select = IO::Select->new($listen);

$|++;
while (1) {
  eval {

    # for all readable sockets
    for my $sock ($select->can_read($timeout)) {

      if ($sock == $listen) {

        # accept a new client socket
        my $client = $sock->accept; # = myAccept($sock);
        if (defined($client)) {
          @{*$client}{qw/sbuf size state/} = ('', 0, 'need_headers');
          $select->add($client);
        } else {
          print "accept error: $!\n";
        }

      } else {

        # already connected client socket
        my $props = *$sock;

        if ($props->{state} eq 'need_headers') {

          # reading incoming request...
          my $read = $sock->sysread(
            $props->{sbuf},$rlen,$props->{size});
          unless (defined $read) {
            $select->remove($sock);
            die "read error: $!\n";
          }
          $props->{size} += $read;

          if (my ($headers) = 
            ($props->{sbuf} =~ /^(.*?)\r\n\r\n(.*)/s)) {{

            # we've finished reading the HTTP header
            use bytes;

            my ($verb, $uri) = ($headers =~ /^(\w+)\s+(\S+)/);
            print "[$verb] [$uri]\n";

            # put any remaining bytes of request back into the buffer 
            # (likely HTTP message body)
            @{$props}{qw/headers verb uri sbuf size/} = (
                $headers, $verb, $uri, 
                substr($props->{sbuf}, length($headers)),
                length($props->{sbuf})
              );

            if (my ($bsize) = 
              ($headers =~ /Content-Length\s*:\s*(\d+)/s)) {
          
              # need to read HTTP message body of length $bsize
              @{$props}{qw/need state/} = ($bsize, 'need_body');

            } else {

              # no HTTP message body follows, done reading request
              @{$props}{qw/size sbuf body state/} = 
                (0, '', '', 'request_done');

            }
          }}

        } elsif ($props->{state} eq 'need_body') {

          # reading body...
          my $size = $props->{need} < $rlen ? $props->{need} : $rlen;
          my $read = $sock->sysread(
            $props->{sbuf},$size,$props->{size});
          $props->{size} += $read;
          $props->{need} -= $read;

          # done reading body (if we've read enough bytes) 
          @{$props}{qw/size sbuf body state/} = 
            (0, '', $props->{sbuf}, 'request_done') 
            unless $props->{need};

        }
      }
    } # -- readable sockets

    # for all writable sockets
    for my $sock ($select->can_write($timeout)) {

      next if $sock == $listen;
      my $props = *$sock;

      # we only want to write to sockets with from which we've read 
      # a full request
      if ($props->{state} eq 'request_done') {

        # request read, build response...
        my $msg;
        
        #
        # YES: I'm aware the path is tainted/insecure.
        #      This is just an example to demonstrate failure.
        #
        if (-f ".".$props->{uri}) {

          # the requested file was found, so...
          # determine mime-type
          my $type = $magic->checktype_filename(
            ".".$props->{uri}) || "text/html";

          # read local file
          open F, "<.".$props->{uri};
          my ($buf, $len) = ('', 0);
          while (my $read = sysread(F, $buf, $fblen, $len)) { 
            $len += $read;      
          }
          close F;

          # will send positive response
          $msg = [200, 'OK', $type, $buf];

        } else {

          # will send negative response
          $msg = [404, 'File Not Found', 'text/html', 'What file?!?'];

        }

        {
          use bytes;

          # construct HTTP response as a string
          $props->{wbuf} =
            'HTTP/1.1 '.
            HTTP::Response->new(
                $msg->[0] => $msg->[1],
                ['Content-Type'   => $msg->[2],
                 'Content-Length' => length($msg->[3]),
                 'Connection'     => 'close'],
                $msg->[3]
              )->as_string;
          $props->{wdone} = 0;
          $props->{wsize} = length($props->{wbuf});
          $props->{state} = 'response_pending';
        }

      } elsif ($props->{state} eq 'response_pending') {

        # writing outgoing response...
        my $size = $props->{wsize} < $wlen ? $props->{wsize} : $wlen;
        my $wrote = $sock->syswrite(
          $props->{wbuf},$size,$props->{wdone});
        unless (defined $wrote) {
          $select->remove($sock);
          die "write error: $!\n";
        }
        $props->{wdone} += $wrote;

        if ($props->{wdone} == $props->{wsize}) {

          # we're done sending the request, ready for another
          # NOTE: IO::Socket::SSL docs say we can't do multiple 
          # requests, but it does work in the absense of 
          # peer cert authentication.
          $props->{wdone} = 0;
          $props->{wsize} = 0;
          $props->{wbuf}  = '';
          $props->{state} = 'need_headers';
          
          # seemingly correct, but blocks listener socket
          #$sock->close(SSL_no_shutdown => 1);  
        }

      }

    } # -- writable sockets

  };
  # print any perl-level exception
  print "uncaught error: $@\n" if $@;

}

# this is something HTTP::Daemon::SSL does...
# seems to be a work-around for premature return 
# from IO::Socket::SSL::accept
sub myAccept
{
  my $self = shift;
  while (1) {
    # I hope this doesn't block too long
    my $sock = IO::Socket::SSL::accept($self);
    return $sock if ($sock || 
      $self->errstr =~ /^IO::Socket[^\n]* accept failed$/);
  }
}
[download]

Patch for IO::Socket::SSL v1.02

SSL peer certificate authentication only seems to work with recent OpenSSL's with the following patch to IO::Socket::SSL (figured this out from talk in SSL related groups on usenet):

--- IO-Socket-SSL-orig.pm    2007-03-05 15:34:15.000000000 +1000
+++ IO-Socket-SSL.pm    2007-03-05 15:40:30.000000000 +1000
@@ -119,6 +119,7 @@
      'SSL_version'   => 'sslv23',
      'SSL_verify_mode' => Net::SSLeay::VERIFY_NONE(),
      'SSL_verify_callback' => 0,
+     'SSL_verify_depth' => 0
     );
      
     # SSL_key_file and SSL_cert_file will only be set in defaults if 
@@ -829,7 +830,7 @@
         |SSL_MODE_ENABLE_PARTIAL_WRITE);
 
 
-    my ($verify_mode, $verify_cb) = @{$arg_hash}{'SSL_verify_mode','S
+SL_verify_callback'};
+    my ($verify_mode, $verify_cb, $verify_depth) = @{$arg_hash}{'SSL_
+verify_mode','SSL_verify_callback','SSL_verify_depth'};
     unless ($verify_mode == Net::SSLeay::VERIFY_NONE())
     {
     &Net::SSLeay::CTX_load_verify_locations
@@ -912,6 +913,7 @@
     };
 
     Net::SSLeay::CTX_set_verify($ctx, $verify_mode, $verify_callback)
+;
+        Net::SSLeay::CTX_set_verify_depth($ctx, $verify_depth);
 
     $ctx_object = { context => $ctx };
     if ($arg_hash->{'SSL_session_cache_size'}) {
[download]

The following are the certificates and keys I'm testing with (which should be located in a directory called 'certs' in the working directory):

certs/my-ca.pem

certs/ca-key.pem

certs/server-cert.pem

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: md5WithRSAEncryption
        Issuer: C=AU, ST=Queensland, L=Brisbane, O=Test Company, OU=Te
+st Company Authority, CN=Test Root CA/emailAddress=rootca@testcompany
+.com
        Validity
            Not Before: Mar  5 05:18:26 2007 GMT
            Not After : Mar  2 05:18:26 2017 GMT
        Subject: CN=www.testcompany.com/emailAddress=test@testcompany.
+com, O=Test Company, OU=Test Server, C=AU, ST=Queensland, L=Brisbane
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    00:cc:c1:df:56:50:28:97:f7:13:6b:13:b1:41:85:
                    99:f4:97:2c:16:2e:0a:22:ad:16:76:98:df:14:a6:
                    c2:a7:a4:54:8a:d7:9a:46:01:a0:b8:24:e6:93:16:
                    57:4a:da:7f:de:08:fe:61:b3:d7:ae:02:ba:3b:80:
                    0e:6c:e6:ee:70:23:1f:a1:66:45:51:d0:cd:83:d0:
                    05:ba:17:86:a9:c8:97:56:1b:a3:c4:24:e4:64:ee:
                    8c:a8:1a:ee:6e:19:7a:cb:cf:2d:8f:c2:a6:1f:b6:
                    53:34:68:fc:ae:89:b2:ca:f7:7e:2c:17:b3:c4:07:
                    a9:8b:20:77:e0:ed:8b:25:2e:13:00:c4:da:bd:29:
                    6a:c6:a3:9a:ed:cb:df:15:e3:d5:8a:c5:59:1e:10:
                    82:16:c9:bf:48:67:3b:3b:41:b0:20:77:93:f3:e9:
                    22:9d:33:68:89:5b:09:0f:82:54:91:50:e4:43:4f:
                    53:3e:67:12:65:5e:3c:8c:a8:c2:1f:b8:27:b2:48:
                    43:20:9e:64:39:d3:cd:c6:73:c7:b4:d7:e1:8b:5a:
                    5c:58:e7:54:ad:fa:1f:56:e0:cd:2a:7f:97:94:86:
                    71:a3:8e:d8:cc:fa:cd:05:58:d4:b7:1a:fb:3f:e1:
                    d7:77:87:b6:22:b9:fa:d8:3a:c2:e9:f5:99:2f:0b:
                    dc:ef
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:90:DF:76:61:6B:50:9F:E9:0B:80:C8:B1:A4:67:18:43:
+51:FD:57:62

            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authenti
+cation, Microsoft Server Gated Crypto, Netscape Server Gated Crypto
            X509v3 Basic Constraints: critical
                CA:FALSE
    Signature Algorithm: md5WithRSAEncryption
        00:56:38:5f:2c:cf:85:8f:68:0b:47:fd:3d:8e:fe:df:04:c1:
        36:14:6f:c6:c7:68:61:f9:64:bd:d8:75:9d:f5:89:7d:31:e6:
        f7:e6:d7:79:8d:19:88:28:2a:9a:6d:84:b8:54:03:68:d3:66:
        ac:6b:eb:f7:b7:32:4e:51:1d:0f:67:c0:d3:c4:60:ae:2b:4a:
        c6:e4:eb:36:98:3f:0b:04:a0:86:73:93:f0:4d:e8:b3:11:e9:
        8d:5b:21:d8:80:ec:69:68:83:1c:01:de:71:1b:57:6c:3e:94:
        71:8b:db:c8:53:ce:0b:27:d3:03:89:59:a4:46:89:31:d6:9d:
        7a:94:81:5d:bb:35:3f:d6:6d:c8:56:60:f5:a6:b9:1e:34:94:
        bd:0d:30:ea:66:26:8a:3c:15:32:34:de:e7:8a:0d:7b:cd:4b:
        26:5a:b4:9f:e1:32:0c:76:78:b9:ad:c5:56:73:6c:49:79:0b:
        35:aa:eb:aa:75:47:80:51:65:a6:1a:be:b9:13:21:7c:b4:57:
        d6:27:b0:62:fd:b4:52:6d:3f:73:cd:f1:78:56:09:a9:1f:93:
        7a:5e:59:12:c0:7d:ea:4a:3a:74:37:24:59:0b:01:4c:06:95:
        e0:23:4e:05:dc:f4:32:7e:fe:dd:1a:11:25:85:2e:e1:ad:0b:
        37:56:c5:67
-----BEGIN CERTIFICATE-----
MIIENDCCAxygAwIBAgIBATANBgkqhkiG9w0BAQQFADCBqzELMAkGA1UEBhMCQVUx
EzARBgNVBAgTClF1ZWVuc2xhbmQxETAPBgNVBAcTCEJyaXNiYW5lMRUwEwYDVQQK
EwxUZXN0IENvbXBhbnkxHzAdBgNVBAsTFlRlc3QgQ29tcGFueSBBdXRob3JpdHkx
FTATBgNVBAMTDFRlc3QgUm9vdCBDQTElMCMGCSqGSIb3DQEJARYWcm9vdGNhQHRl
c3Rjb21wYW55LmNvbTAeFw0wNzAzMDUwNTE4MjZaFw0xNzAzMDIwNTE4MjZaMIGl
MRwwGgYDVQQDExN3d3cudGVzdGNvbXBhbnkuY29tMSMwIQYJKoZIhvcNAQkBFhR0
ZXN0QHRlc3Rjb21wYW55LmNvbTEVMBMGA1UEChMMVGVzdCBDb21wYW55MRQwEgYD
VQQLEwtUZXN0IFNlcnZlcjELMAkGA1UEBhMCQVUxEzARBgNVBAgTClF1ZWVuc2xh
bmQxETAPBgNVBAcTCEJyaXNiYW5lMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEAzMHfVlAol/cTaxOxQYWZ9JcsFi4KIq0WdpjfFKbCp6RUiteaRgGguCTm
kxZXStp/3gj+YbPXrgK6O4AObObucCMfoWZFUdDNg9AFuheGqciXVhujxCTkZO6M
qBrubhl6y88tj8KmH7ZTNGj8romyyvd+LBezxAepiyB34O2LJS4TAMTavSlqxqOa
7cvfFePVisVZHhCCFsm/SGc7O0GwIHeT8+kinTNoiVsJD4JUkVDkQ09TPmcSZV48
jKjCH7gnskhDIJ5kOdPNxnPHtNfhi1pcWOdUrfofVuDNKn+XlIZxo47YzPrNBVjU
txr7P+HXd4e2Irn62DrC6fWZLwvc7wIDAQABo2cwZTAfBgNVHSMEGDAWgBSQ33Zh
a1Cf6QuAyLGkZxhDUf1XYjA0BgNVHSUELTArBggrBgEFBQcDAQYIKwYBBQUHAwIG
CisGAQQBgjcKAwMGCWCGSAGG+EIEATAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEB
BAUAA4IBAQAAVjhfLM+Fj2gLR/09jv7fBME2FG/Gx2hh+WS92HWd9Yl9Meb35td5
jRmIKCqabYS4VANo02asa+v3tzJOUR0PZ8DTxGCuK0rG5Os2mD8LBKCGc5PwTeiz
EemNWyHYgOxpaIMcAd5xG1dsPpRxi9vIU84LJ9MDiVmkRokx1p16lIFduzU/1m3I
VmD1prkeNJS9DTDqZiaKPBUyNN7nig17zUsmWrSf4TIMdni5rcVWc2xJeQs1quuq
dUeAUWWmGr65EyF8tFfWJ7Bi/bRSbT9zzfF4VgmpH5N6XlkSwH3qSjp0NyRZCwFM
BpXgI04F3PQyfv7dGhElhS7hrQs3VsVn
-----END CERTIFICATE-----
[download]

certs/server-key.pem

certs/client-cert.pem

You may need to convert this to p12 format for import into your browser.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2 (0x2)
        Signature Algorithm: md5WithRSAEncryption
        Issuer: C=AU, ST=Queensland, L=Brisbane, O=Test Company, OU=Te
+st Company Authority, CN=Test Root CA/emailAddress=rootca@testcompany
+.com
        Validity
            Not Before: Mar  5 05:22:54 2007 GMT
            Not After : Mar  4 05:22:54 2009 GMT
        Subject: CN=Test Client/emailAddress=client@testcompany.com, O
+=Test Company, OU=Test Client, C=AU, ST=Queensland, L=Brisbane
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (1024 bit)
                Modulus (1024 bit):
                    00:c2:b3:fc:60:97:5b:65:07:d5:36:f5:e5:2a:cb:
                    6c:d8:30:fe:17:83:6a:84:0f:95:2a:a7:3d:b4:61:
                    fd:a6:cb:87:7f:4b:44:6a:d0:a3:21:86:4c:ba:6c:
                    91:6e:04:8a:92:34:03:cc:fa:c0:73:6c:7e:b0:27:
                    84:a5:a2:4a:cf:cb:75:c0:06:f6:8d:af:94:6d:a5:
                    7f:1a:9a:da:07:49:dc:eb:00:c6:03:46:70:9b:7d:
                    2e:f4:90:94:f1:56:05:8f:02:26:43:58:58:07:2d:
                    8f:da:81:24:2e:37:83:af:5a:50:58:ae:33:f7:d8:
                    24:de:74:99:0a:9d:33:6b:eb
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                email:client@testcompany.com
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier: 
                keyid:90:DF:76:61:6B:50:9F:E9:0B:80:C8:B1:A4:67:18:43:
+51:FD:57:62

            X509v3 Extended Key Usage: 
                TLS Web Client Authentication, E-mail Protection
    Signature Algorithm: md5WithRSAEncryption
        16:49:c0:ca:58:e5:ef:9b:b2:f7:10:c6:12:1d:e5:eb:13:ba:
        f5:d4:2c:50:91:16:71:0e:70:b5:3a:db:7e:7e:3c:c6:2e:66:
        4d:fb:85:fa:a3:ef:a7:4a:b5:0e:e4:c9:a3:b0:d3:1d:46:3c:
        96:e5:65:3a:e2:ad:65:7d:9a:12:6d:47:67:65:0a:b9:92:48:
        fa:96:6a:8b:51:58:1f:79:1e:5b:af:89:b8:c9:92:73:c0:88:
        cb:a5:76:a7:45:f3:e0:48:6d:d5:69:f6:74:aa:d3:39:e6:c8:
        fc:d1:89:2a:e9:bf:88:8c:0b:5c:e8:d7:fa:3d:74:21:7d:c8:
        d5:11:d2:63:ae:3e:00:48:36:a7:a8:41:e8:08:d3:81:0b:80:
        25:42:aa:df:f7:c6:4d:17:a0:b0:0f:55:74:0c:8b:f1:b9:84:
        1f:75:d5:3d:1e:45:14:06:b2:e1:33:4d:c1:6a:a9:b4:ee:7c:
        bf:49:cd:c3:36:04:47:41:b2:87:36:00:29:59:28:07:fe:74:
        70:24:d4:d8:ec:3c:64:d6:5c:1e:81:c1:32:b7:86:36:12:db:
        6b:0e:3a:e5:54:a0:ef:d9:c1:8c:98:5c:83:c6:02:d2:f3:49:
        c1:91:55:6a:c9:4c:7f:c6:57:c7:e1:8d:bc:7e:78:30:ab:95:
        4d:b0:11:63
-----BEGIN CERTIFICATE-----
MIIDtjCCAp6gAwIBAgIBAjANBgkqhkiG9w0BAQQFADCBqzELMAkGA1UEBhMCQVUx
EzARBgNVBAgTClF1ZWVuc2xhbmQxETAPBgNVBAcTCEJyaXNiYW5lMRUwEwYDVQQK
EwxUZXN0IENvbXBhbnkxHzAdBgNVBAsTFlRlc3QgQ29tcGFueSBBdXRob3JpdHkx
FTATBgNVBAMTDFRlc3QgUm9vdCBDQTElMCMGCSqGSIb3DQEJARYWcm9vdGNhQHRl
c3Rjb21wYW55LmNvbTAeFw0wNzAzMDUwNTIyNTRaFw0wOTAzMDQwNTIyNTRaMIGf
MRQwEgYDVQQDEwtUZXN0IENsaWVudDElMCMGCSqGSIb3DQEJARYWY2xpZW50QHRl
c3Rjb21wYW55LmNvbTEVMBMGA1UEChMMVGVzdCBDb21wYW55MRQwEgYDVQQLEwtU
ZXN0IENsaWVudDELMAkGA1UEBhMCQVUxEzARBgNVBAgTClF1ZWVuc2xhbmQxETAP
BgNVBAcTCEJyaXNiYW5lMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDCs/xg
l1tlB9U29eUqy2zYMP4Xg2qED5Uqpz20Yf2my4d/S0Rq0KMhhky6bJFuBIqSNAPM
+sBzbH6wJ4SlokrPy3XABvaNr5RtpX8amtoHSdzrAMYDRnCbfS70kJTxVgWPAiZD
WFgHLY/agSQuN4OvWlBYrjP32CTedJkKnTNr6wIDAQABo3MwcTAhBgNVHREEGjAY
gRZjbGllbnRAdGVzdGNvbXBhbnkuY29tMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgw
FoAUkN92YWtQn+kLgMixpGcYQ1H9V2IwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsG
AQUFBwMEMA0GCSqGSIb3DQEBBAUAA4IBAQAWScDKWOXvm7L3EMYSHeXrE7r11CxQ
kRZxDnC1Ott+fjzGLmZN+4X6o++nSrUO5MmjsNMdRjyW5WU64q1lfZoSbUdnZQq5
kkj6lmqLUVgfeR5br4m4yZJzwIjLpXanRfPgSG3VafZ0qtM55sj80Ykq6b+IjAtc
6Nf6PXQhfcjVEdJjrj4ASDanqEHoCNOBC4AlQqrf98ZNF6CwD1V0DIvxuYQfddU9
HkUUBrLhM03Baqm07ny/Sc3DNgRHQbKHNgApWSgH/nRwJNTY7Dxk1lwegcEyt4Y2
EttrDjrlVKDv2cGMmFyDxgLS80nBkVVqyUx/xlfH4Y28fngwq5VNsBFj
-----END CERTIFICATE-----
[download]

Update: peer cert auth problem has been resolved.
Please feel free to make architectural suggestions, the more options the better.

-David.

Comment on Multiplexing HTTPS server, peer cert authentication problem. Select or Download Code

Replies are listed 'Best First'.
Re: Multiplexing HTTPS server, peer cert authentication problem. by Thelonius (Priest) on Mar 05, 2007 at 17:51 UTC
One possibility, as Moron implies, is to write a forking server, possibly a pre-forking server. You might need to use Cygwin to get forking to work correctly on Windows. However, your approach might work with a little more code. From the documentation for IO::Socket::SSL Note that if start_SSL() fails in SSL negotiation, $socket will remain blessed in its original class. For non-blocking sockets you better just upgrade the socket to IO::Socket::SSL and call accept_SSL or connect_SSL on the upgraded object. To just upgrade the socket set B<SSL_startHandshake> explicitly to 0. If you call start_SSL w/o this parameter it will revert to blocking behavior for accept_SSL and connect_SSL. My interpretation, without actually trying this is: my $acceptsock = $sock->accept; my $sslaccept = IO::Socket::SSL->start_SSL($acceptsock, {SSL_startHandshake => 0, SSL_use_cert => 1, SSL_verify_depth => 1, SSL_verify_mode => 0x03, }); $select->add($sslaccept); @{$sslaccept}{qw/sbuf size state/} = ('', 0, 'handshake'); # then go back to your select() # Later, when the select() returns the $sslaccept socket, call if ($sock->{state} eq 'handshake') { my $sslclient = $sock->accept_SSL(); if (defined($sslclient)) { # success! # advance the state of socket to connected, etc. *$sock->{state} = 'need_headers'; } elsif ($SSL_ERROR == SSL_WANT_READ \|\| $SSL_ERROR == SSL_WANT_READ) { # just do another select, then repeat call to accept_SSL # no code needed here, I think } else { # Otherwise, the connection has failed. $select->remove($sock); $sock->close(); # maybe log it } } [download] You will probably need to use the three argument select instead of can_read(): `my ($readsocks, $writesocks, $errsocks) = IO::Select::select($select, $select, $select);` [download]	[reply] [d/l] [select]
Re^2: Multiplexing HTTPS server, peer cert authentication problem. by erroneousBollock (Curate) on Mar 06, 2007 at 01:09 UTC
I'm not sure I understand. I do understand that you're saying I should split the (SSL) accept() into a non-SSL accept() and a start_SSL(). What I don't understand is what problem that solves :) From what I have read, SSL_startHandshake makes sure that start_SSL is non-blocking (doesn't re-bless socket until it succeeds or fails). So, is your suggestion that the peer certificate authentication goes awry somehow, because I'm still blocking? -David.	[reply]
Re^3: Multiplexing HTTPS server, peer cert authentication problem. by Thelonius (Priest) on Mar 06, 2007 at 18:10 UTC
Well, I am suggesting it because the documentation recommends it. I tried it out and it seems to work, with one change, that it needs `SSL_server => 1` here: `my $sslaccept = IO::Socket::SSL->start_SSL($acceptsock, {SSL_startHandshake => 0, SSL_server => 1, SSL_use_cert => 1, SSL_verify_depth => 1, SSL_verify_mode => 0x03, });` [download] Otherwise it will try to authenticate as a client. Here's a restructured program: Read more... (7 kB)	[reply] [d/l] [select]
Re^4: Multiplexing HTTPS server, peer cert authentication problem. by erroneousBollock (Curate) on Mar 07, 2007 at 01:59 UTC
Re: Multiplexing HTTPS server, peer cert authentication problem. by Moron (Curate) on Mar 05, 2007 at 18:33 UTC
Going back to the original problem yet again, you have a daemon that accepts connections, but misses new requests while servicing the last one. Next idea is: Following the way, functionally, apache solves the same problem, you would have a master daemon that spawns eight eager daemonets all grabbing connections and being busy servicing them - not for long because they'll spawn another process to actually handle the request, e.g. Perl if a cgi program is allocated to the URL, and then go back to grabbing the next request. That way, translating back to your situation, if one daemonet is busy, there is always another one available to do the accept call. The looping required for each daemonet would be something like: `while ($request = $listen->accept()) { # handle request ... BUT ...read on }` [download] Instead of spawning new processes to handle requests like apache does or forking or threading, the daemonets could write the request to a fifo. That way you can separate the asynchronous process of collecting and filing requests from the synchronous process of reading off the queue and servicing them - it is generally a good idea not to mix asynchronous and synchronous processing in the same process (this being what seems to be at the root of it all) - in fact that would be an oxyMoron ;) -M Free your mind	[reply] [d/l]
Re^2: Multiplexing HTTPS server, peer cert authentication problem. by erroneousBollock (Curate) on Mar 06, 2007 at 01:44 UTC
Ah, so you're talking about multiple listeners for the same socket - "bound" before the inital fork(), where only one process actually succeeds to accept() for a given incoming request ? Does that work with threads ? :) I only ask because I've looked into the magic that Threads.pm does to clone (copy) non-shared variables into new threads... specifically, that does bad things to the self-glob-ref thing that IO::Socket::SSL does internally. -David.	[reply]
Re^3: Multiplexing HTTPS server, peer cert authentication problem. by Moron (Curate) on Mar 06, 2007 at 10:40 UTC
If you are sticking with threads, you can pass objects to threads using the Thread::Queue module. In fact that is also the recommended (big Camel book) way to synchronise the handling of asynchronous requests under those circumstances, although if you use it to meet both requirements you'd need N+1 queues where N is the number of startup threads - one to synchronise requests to the request-handling synchronous daemon and one for each startup daemonet to receive object data from the initialising parent. And if the threads approach remains too troublesome, you could always spawn the daemonets with something like for (1..$nrDmnts) { my $pid = open my $ph, "\|-" ... } instead and send them non-shared data across the pipe. -M Free your mind	[reply]
Re^4: Multiplexing HTTPS server, peer cert authentication problem. by erroneousBollock (Curate) on Mar 06, 2007 at 12:18 UTC
Re^5: Multiplexing HTTPS server, peer cert authentication problem. by Moron (Curate) on Mar 06, 2007 at 12:25 UTC
Some notes below your chosen depth have not been shown here
Re: Multiplexing HTTPS server, peer cert authentication problem. by Moron (Curate) on Mar 05, 2007 at 16:56 UTC
Were forks as well as threads eliminated from the original state of the problem? -M Free your mind	[reply]
Re^2: Multiplexing HTTPS server, peer cert authentication problem. by erroneousBollock (Curate) on Mar 05, 2007 at 17:23 UTC
In my testing (on Windows) fork() ended up using Threads.pm, which made IO::Socket::SSL break. I am using threads, just not to do communication with clients. The package which contains the multiplexing https server has a reference to a 'dispatcher' object with methods like: * dispatcher->dispatch(headers,body) * dispatcher->pending_responses() * dispatcher->retrieve_response() The dispatcher runs in a separate thread - and manages dispatch of "work" to a pool of worker threads via Thread::Queue::Any. -David.	[reply]
Re: Multiplexing HTTPS server, peer cert authentication problem. by perrin (Chancellor) on Mar 05, 2007 at 18:43 UTC
I expect you have a reason for this, but I'm curious: why not use apache with mod_perl or FastCGI?	[reply]
Re^2: Multiplexing HTTPS server, peer cert authentication problem. by erroneousBollock (Curate) on Mar 06, 2007 at 01:39 UTC
Yes, I do have some reasons ;) The customer's constraints are that: the server must be a standalone Windows executable the same executable must be capable of running as a Windows NT service executable the server should not depend on other software My constraint is that the same codebase (or close to) runs on Linux, so that I may eventually convince the customer to move off Windows. More background: Basically, this thing is a "Webservice Toolkit" specialised for this particular customer. The service listens on a single port for HTTPS requests in SOAP or JSONRPC format. An authentication module is configured to take information from the request headers, body or SSL peer certificate attributes in order to check whether the client may proceed. The authentication information is then used to retieve "user context" from a database for the current client. That context is visible from exposed "service classes". An (Apache-like) chain of Handler modules is configured for the service such that certain urls are mapped to a combination of an exposed "service class" and a service "handler type". (eg: https://service:2222/WholesaleBillingSOAP is configured to expose the Wholesale::Billing class via SOAP.) Additional handler types exist for non-programatic access to the service. For this customer, the main user interface is a Macromedia Flex application (browser-based) which is served from the service at https://service:2222/sample/Wholesale.html. That Flex application reads an XML configuration file (also served by the service) to find out where the service is running, then talks to the service via SOAP. Finally, the service includes asynchronous functionality that can be requested by a SOAP or JSONRPC client, which will run for a potentially long time (ie: reports which take minutes to run). Those clients can request a progress report or the final results of the long-running job. I hope that clears up some of my motivation. -David.	[reply]
Re^3: Multiplexing HTTPS server, peer cert authentication problem. by perrin (Chancellor) on Mar 06, 2007 at 19:12 UTC
These requirements seem pretty arbitrary. Is your server, which depends on perl, SSL libs, and a bunch of modules, really more "standalone" than Apache and some perl code? I suspect the real meaning is something like "must be easy to install and not have any licensing fees." There are some nice click-and-drool installers that will put Apache, mod_perl, and mod_ssl on a Windows machine, and you could add more to them. Maybe I'm just overestimating the difficulty of writing a reliable HTTP server. If this is a controlled environment with specific client browsers, it may be easier than usual. You could look at some of the existing non-blocking servers like AxKit2 for help. I haven't seen a working SSL plugin for it though. There's also some POE stuff that I believe does work with SSL. Is it really that important to use non-blocking I/O? There are simple, well-established techniques for handling long-running processes from CGI that can be used for most forking/threading situations. You essentially spawn a thread to handle the job and have it write progress updates somewhere that you can grab them for display.	[reply]
Re^4: Multiplexing HTTPS server, peer cert authentication problem. by erroneousBollock (Curate) on Mar 07, 2007 at 01:06 UTC
Re^5: Multiplexing HTTPS server, peer cert authentication problem. by perrin (Chancellor) on Mar 07, 2007 at 01:33 UTC
Re: Multiplexing HTTPS server, peer cert authentication problem. by erroneousBollock (Curate) on Mar 07, 2007 at 02:13 UTC
Hi monks, Thelonius figured out that the problem had to do with the SSL handshake. From my debugging it looks like the peer cert auth adds another 2 round-trip messages during the SSL handshake. It turns out that in the code I posted in the OP, I was not ready to read/write at the appropriate moments during the handshake (and therefore, my state machine was incorrect). It also appears that calling accept() in IO::Socket::INET, followed by start_SSL() in IO::Socket::SSL is doing something slightly different that just calling accept() in IO::Socket::INET with equivalent arguments. I'll post more if I find out what that is. UPDATE: It was not clear to me from the docs that if you wish to convert a connected IO::Socket::INET to connected IO::Socket::SSL, there are actually 3 steps: call accept() in IO::Socket::INET, to get a connected socket. call start_SSL() in IO::Socket::SSL, passing the connected socket and the param: SSL_startHandshake = 0. call accept_SSL() on the "converted" connected socket until the result is defined or there is some non-useful error in $SSL_ERROR. Thank-you monks for your help and your patience (wrt my long-windedness ;). -David.	[reply]


Perl: the Markov chain saw
	PerlMonks