|
I'm being encouraged to offer image-hosting for members of my site.
I can not find any existing software to fulfill my needs. The only image-hosting applications I've found are those geared toward offering simple galleries in which limitations are only tracked and imposed when access is made directly through the application's interface.
Required Features:
Upload, rename, delete, view and categorize images.
Create account levels, allowing limitations on number of images, maximum size of images, maximum size of all combined images, number of views per image and combined views.
Track and impose the above limits.
I've only devised two solutions.
The first is to use ModRewrite and when a key path exists in the URL, kick off a script to update an SQL database to count the hit and calculate the bandwidth usage. If any limits are exceeded, the image is not displayed and access is denied. The downside is that ModRewrite, an SQL UPDATE/INSERT and an SQL SELECT for every image serged is a massive amount of overhead.
The second solution is to have a small script that parses the Apache log files on a regular basis (hourly?) and updates the database to calculate various usage stats. When limits have been exceeded, the script would back the original image up and replace it with a very small (few bytes) PNG containing an error message.
If anyone has input on this - or even better, happens to know of an existing application that I somehow could not find - please inform me!
|
•Re: Image Hosting Application
by merlyn (Sage) on Jul 20, 2004 at 16:53 UTC
|
Through proper use of a mod_perl AuthzHander and Apache::DBI, you should be able to update statistics in real time and get reasonable performance from any solid transaction-based database (hint: PostgreSQL or Oracle, but probably not MySQL).
| [reply] |
|
|
I've not used mod_perl in an application yet - though I understand it. This is mostly due to the fact that the main service I run on my site is based on code I wrote awhile back that I am not positive would run without consistancy problems mucking everything up. I suppose a simple image-hosting script could be devised well enough to take the mod_perl plunge.
And yes, I do code with use strict; use warnings; use diagnostics; on.
I've wanted to move to mod_perl for two years now, but am apprehensive as it's a major change and the only solid way to know your code will perform reliably in a mod_perl situation is to test it with a lot of active users, which I don't have a way of doing short of putting it on the production server and crossing my fingers. :)
| [reply] |
|
|
Using mod_perl for an authz handler is independent of whatever old code you have at the mod_cgi level during the content phase. Feel free to migrate at your desired speed.
However, the larger question is, how are you expected to do development when you don't have a valid test system including a load generator? I'd solve that problem first before worrying about a new bunch of code.
| [reply] |
|
|
|
|
| [reply] |
|
|
Yes, but from what I understand, when you start using InnoDB, you sacrifice the only advantage MySQL has, which is speed and reliability for non-transactional database accesses. So if you want a fast, reliable transactional database, even MySQL with InnoDB doesn't cut it. Hence, my recommendation for PostgreSQL or Oracle, depending on your budget.
| [reply] |
Re: Image Hosting Application
by perrin (Chancellor) on Jul 20, 2004 at 16:56 UTC
|
There are several existing Apache modules for limiting bandwidth, as well as some mod_perl stuff on CPAN. No need to start from scratch. | [reply] |
|
|
I thought the bandwidth limiting/shaping stuff was tied to IPs and/or virtual domains. I'd need to have the limiting applied to users that would not physically exist anywhere other than a record in a table in Postgres.
If this is possible - then it sounds like a great way for me to go. I'll persue this through a refreshed browse through apache.org. Thanks. :)
| [reply] |
|
|
For truly custom stuff that can't be fit into the existing C modules, use one of the Apache:: modules on CPAN and modify it to meet your needs.
| [reply] |
Re: Image Hosting Application
by BUU (Prior) on Jul 20, 2004 at 22:41 UTC
|
Since you seem to have two main concerns, efficiency and testability, let me try to adress them in turn.
First off, efficiency. As the mantra goes, premature optimization is the root of all evil. In other words even if you *think* something might be slow, benchmark it and find out! If, for some reason, a sql query and a mod_rewrite handler call is excessive over head for sending an image (for some reason, I doubt it, but maybe..), then you have several options.
My first option would be to maintain a hash in memory (Using one of the IPC::Share type modules) that all of your scripts can accesss to get current bandwidth usage. Thus removes the sql transaction time and should speed it up.
The second component of this system would be a daemon that regularly polls the in-memory hash and writes it's data to a database. Just make sure you use locking.
Only problem I see here is say, the daemon checks every hour for new data, and your machine crashes 59 minutes after the daemon last run, you could lose up to 59 minutes worth of data transfer. I don't really see this as a problem personally, since from the type of application you describe an extra dozen or hundred "free" requests shouldn't hurt you or anyone else, and may make a decent "bonus" since the machine did crash.
(On a slightly different tack however, have you considered, instead of bandwidth limits and so fort, only allowing the images to be shown on your server and then just not worrying about bandwidth and so forth?)
Your second concern, testability, I have two thoughts. The first thought is limited to the situation at hand, in which testing should be simple. Implement whatever you want, add some pictures and some limits, then just have AB or something similar generate a few thousand requests for a gif. Theres your testing. Not hard.
As for testing in general, the idea that came to mind was to take a snapshot of the data for your app at a particular time, then start recordding http requests (perhaps by reading them from the log file, whatever). Once you had say, a days worth of requests, you could use the snapshot-data you stored previously to recreate the conditions on your test machine, then play back the http requests. You could even play them back at a faster rate, say a day in 3 hours or something to generate higher load. | [reply] |
Re: Image Hosting Application
by gaal (Parson) on Jul 20, 2004 at 17:12 UTC
|
Take a peek at FotoBilder, a cousin project of LiveJournal. It hasn't entered large scale production just yet, but once it does -- and this should happen soon -- it will get lots of scalability love.
Perl, open source, mod_perl based. | [reply] |
Re: Image Hosting Application
by markjugg (Curate) on Jul 20, 2004 at 18:08 UTC
|
I'm currently working on tackling part of this problem with CGI::Uploader. It's a work in progress, but may give you some ideas. Right now I'm re-factoring it (and breaking the API) to make it less centered around images and better handle other kinds of uploads.
Feedback welcome.
| [reply] |
|
|
