• DBA
• Sysadmin

Unless you have personally been a DBA or sysadmin on a large project, you will mess it up. This isn't optional. There are a number of critical FAIL-if-you-don't-take-into-account considerations that a programmer will never think of, because that's not their job. I am both a programmer and a DBA, but I'm not a sysadmin, so I make sure I have a list of very good sysadmins I can call when I need to. Just as I'm on that list of DBAs for many monks. Unless you have done this, everything else is completely and utterly worthless.

I'm saying that as someone who's been DBA and developer on a number of large web applications written in Perl using a MySQL backend. If you would like some further advice on the matter, feel free to contact me either here on at rkinyon@cpan.org

Beyond that, it's the basics. Pick a web framework, an ORM, webserver, and application server. I personally prefer Catalyst, DBIx::Class, Apache, and FCGI. I know smart people who like Jifty, Jifty, Apache, and mod_perl. There's a number of choices. The key is to pick one and learn it inside and out.

As for laying out the app, screw the documentation crap. Build it with tracer bullets and week-long deliverables. If you can't break a piece down to week-long chunks, look again.

I appreciate your frank response dragonchild. This is obviously something I need to bring up to my client, so what is the recommended course of action as far as telling my client they need to hire someone with those skill sets as well?

Also, what do you mean by break it down into "week long chunks"? And how do you feel about using a pre-existing software package and modifying it? I had been thinking about nearly copying and pasting the logic into a Catalyst application and modifying it from there, because i KNOW how to set up Catalyst with FastCGI and use DBIx::Class with it, and I KNOW Catalyst can handle the load.

Once again, I really appreciate the matter of fact advice

meh.

You'll likely get a lot of good advice from your questions, but the truly wise are those that seek answers to the questions they didn't even know to ask.

That would be your question #4.

• Isolated test, QA & development environments
• QA - unit/regression test management & load testing
• Source code control - consistent procedures for fork/merge, etc.

Good points, I would definitely recommend looking into Selenium (http://www.openqa.org/). It's a tool that lets you record a browser session and then export it as a test (you can export to a number of languages, including perl). The tests can then be rerun at any time to make sure you didn't break anything. It's not only good for automated web interface testing, but also for things like populating a new install with data so you can run your tests against populated systems.

Dude, I love this question and hope you get lots of great answers (I'd love to learn from it too, this is just about where I'm finding myself at this point) but dragonchild is speaking gospel. If you have to ask these questions, you're probably not ready to do the project (at least not alone). I'd say, go slowly and try to emulate other successful, *similar* architectures.

As far as books go, I just picked up Building Scalable Web Sites and Scalable Internet Architectures a couple of weeks ago. I find the first to be the worst O'Reilly title I own -- I'm only halfway through and having trouble forcing myself to read the rest -- packed with gems like: "here's a great technique, but it doesn't scale" and "you don't have to consider race conditions in your architecture as long as they're pretty rare." Scalable Internet Architectures I like much better. A good road map for real definitions and real planning. Flip through it at the bookstore and pick up a copy if you like it.

There is no such thing as a web application, there is only a web interface. Personally- this is what I do.. If I have a FastFood application, my tests.t, FastFood.pm, bin/fastfood, FastFood/Base.pm, etc code does nothing at all with web anything.
I develop FastFood in a terminal environment. Test it all out etc.
Once I am somewhat happy with that, I will start work on FastFood::WUI (Web User Interface).

I have come to great affection for CGI::Application, so my WUIs are CGI::Application based interfaces to ..
whatever- in this case; FastFood.
Thus, my fastfood.cgi only uses FastFood::WUI, not FastFood, FastFood::Base, none of that.

If you want to use CGI::Application- one word of advice; use as many of the plugins as possible, if they do what you want- even if what the way they do it seems counter-intuitive to you. I've found that the plugins seem at first to do some unneeded stuff.. but when I proceed with it.. things make more sense. Almost as if.. the people that came before me knew what they were doing when they suggested these ways of working.. go figure!

Compared to SQLite, in my experience- mysql is slow at inserts and fast on queries; consider doing any heavy lifting, operations.. cpu intense whatevers.. all offline, that is.. drop a crontab entry.

Keep focus on the web interface *as the web interface* and NO more. Any operations, db interactions etc.. Code all of that in your base modules. If the code is not about taking user input or displaying output- it does not belong in the "web app", the web user interface, the WUI.

My final suggestion would be to look up Conway's book on Perl Best Practices, although you probably have it already. He talks about doccumentation in there somewhere, I believe he also suggests writing the docs first, then the code. One very disciplined way to work is to write the tests first, as how you would like to interact with the non existing API- and then write your API so that the tests do not fail- I use Test::Simple- very sexy.

Compared to SQLite, in my experience- mysql is slow at inserts and fast on queries; consider doing any heavy lifting, operations.. cpu intense whatevers.. all offline, that is.. drop a crontab entry.

Spoken as a true programmer ... as in, someone who isn't a DBA. The SQLite DBMS is a great toy. It can be a useful development aid. But, unlike a RDBMS, SQLite doesn't enforce relationships, types, or anything that one would need to ensure data integrity. It doesn't have transactions (ACID or not), proper indices, and a lot of other stuff. In other words, the moment your application started being real, SQLite would do anything from fall over immediately (best case) to corrupt your data without you knowing it (worst case).

While I prefer MySQL (using the InnoDB tabletype only), PostgreSQL, Oracle, DB2, Sybase / SQL*Server ... anything that's ACID-transaction compliant, provides proper foreign keys, indices and the rest of it is necessary for a proper application. This isn't optional.

The rest of what you said - the WUI thing is a really cool concept that I think I'll be grabbing. But, data is the foundation of an application - treat it properly or it will beat you senseless.

---

My criteria for good software:

1. Does it work?
2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

http://london.pm.org/pipermail/london.pm/Week-of-Mon-20071008/010290.html

The original poster started getting stroppy with the people helping him after a while, and the whole thing degenerated pretty quickly which is, IMHO, also worth a read.

f

The first thing to do ... the very first thing! ... is to determine exactly what you need to do, and to lay out an approximate timeline for doing it.

This is not simply a "technical" task. In other words, "MySQL" and "clusters" and so-forth are merely parameters of the project's implementation. Their presence creates yet-another requirement that you must schedule for, namely: determining how you're going to deal with those issues. That will mean research, and you can't just "seat of your pants" any research-related issue. You must formally investigate them and formally come up with an answer to every one.

In any web-application, it's axiomatic that "the user neither knows nor cares how it's done." The user, and therefore your custome, cares only that it does or does-not meet their expectations for the site. An "unspoken" expectation can kill you.

As you proceed through first the planning-stage of your process and then its execution, make definite milestones and then hit them. Don't allow change to come into the project willy-nilly: if the executive vice-president wanders into your office one day with a new starry-eyed idea that he just lifted from iTunes or Google or what-have-you, don't let him or her "swish!" that into the project or your schedule.

Many open-source projects swear by the maxim "build early and often." Ditto: "test." That's a very good principle to adopt and stick-to.

Insist on weekly detailed meetings with your boss or client. What you view as "being well-compensated" is, to someone else, "bloody expensive!" Whatever you do, always remember that! Review your progress, discuss any and all issues, and keep them thoroughly in-the-loop. You will be judged, and you will be paid or not-paid, more by "expectations" than by "results."

Learn from others highscalability.com. I found it very interesting.