Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re: Scalable application with perl

by jasz (Initiate)
on Apr 26, 2016 at 14:29 UTC ( [id://1161558]=note: print w/replies, xml ) Need Help??


in reply to Scalable application with perl

Here are the precise details.

well, to be precise, around 12k users connect to the website for 1.5 hrs just once in a day concurrently for taking the test. Consider it a event happening about 20 times in a year

The, application framework is built with bootstrap...nothing much of aesthetic

each user will be connected and the session is generated and used for the time line mentioned above

the data for (multiple choice) questions retrieved from db and the user submit a choice that is stored in the users table (assuming around 12k rows and 120 columns as nothing much)

with this requirement, like in uwsgi, the concurrency using threads vs process and how many r needed

how db handlers in the script for each user should be implemented with threads or process?

approximately what will be the system requirements on the back end to deal with the situation?

if much of code is implemented on client side, the security checks in the code to be implemented are ?

security rights is not a problem as the data out of this app is under trial

Replies are listed 'Best First'.
Re^2: Scalable application with perl
by Your Mother (Archbishop) on Apr 26, 2016 at 16:29 UTC

    Sounds like a fairly simple problem after all maybe. 12,000 users, but what are they doing during that 90 minutes? One GET and one POST each? Dozens? A hundred? Can users undo answers or backpage? Issues like this can change a simple problem into a terribly complicated one quickly.

    Vanilla nginx can serve pre-built or cached GETs of form pages to the tune of 12K per second just fine if you have the bandwidth. The incoming POSTs have to go through the application though and you won't know how fast that is until you build or at least prototype it. uwsgi has deep and powerful controls for how many worker processes to run and can even make changes dynamically. nginx has an experimental perl embed module now too which could be amazing (I haven't tried it yet). Anyway if you have the RAM and a halfway decent CPU I imagine you'll be able to handle the use you're talking about unless a lot of the users are banging on submit the whole time.

    Tips–

    • Prebuild and cache (probably just as files) all forms. Serve them statically or from webserver memory; both are terribly fast in nginx. There are many Perl templating engines to accomplish this. Text::Xslate is a very fast one (the right choice for "live" service) and Template::Toolkit is a very expressive and well known, documented, and extended one.
    • If the UI is complicated (users can backup and change answers, for example) save state in the browser with JS localStorage. It's a mild hassle to duplicate the form validation in the Perl and the JS but it will keep the webserverapp load zero till submit. If you split an app like this, good testing becomes crucial to avoiding nasty surprises.
    • Investigate embedding the code in nginx but uwsgi probably has the dynamic load options you need.
    • Extra clock savings: don't touch the DB at all during the exams. Dump the validated submissions as single field delimited strings to a flat file or NoSQL or similar. Whatever is fastest and puts lightest load on the machine. This is not my forté, I have no direct recommendation today. When the exam is over, transform and load the flat data to the DB. Then you can grade and run reports from there.
      Extra clock savings: don't touch the DB at all during the exams. Dump the validated submissions as single field delimited strings to a flat file or NoSQL or similar.
      Excellent tip. You could use a key/value store like DB_File to save each submission. If you can construct the form "action" attribute to give it a unique path (perhaps using Javascript on the client side) then you would not even have to parse the query arguments:
      <form action="myapp/user1/question1"> <form action="myapp/user1/question2"> <form action="myapp/user2/question1">
      Then use $ENV{'PATH_INFO'} as the key ('/user1/question1'), and read from STDIN directly to get the raw value to store.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1161558]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others examining the Monastery: (7)
As of 2024-04-19 08:37 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found