|
kiat has asked for the wisdom of the Perl Monks concerning the following question:
Hi,
I've a couple of questions regarding the processing of a flat file using a perl cgi program. The program needs to edit, append and delete entries in the file. My questions are:
1) Does the fact that the database is a flat file make the processing inherently error prone?
2) If the flat file gets very large, does it increase the probability of an error during processing?
3) Does using a structured database (database files created using, say, MySql) make the processing safer?
I hope someone can enlighten me on the above. Thanks in anticipation :)
kiat |
(Ovid) Re: perl cgi and flat textfile
by Ovid (Cardinal) on Dec 07, 2001 at 05:18 UTC
|
I am tempted to simply say "yes" to all three of your questions, but that really wouldn't be fair. It really depends on your coding skill and how much extra work you want to put into it. Using one-liners, here are basic steps:
# deleting entries
perl -ni -e 'print if $_ !~ /old_stuff/' somefile.txt
# editing entries
perl -pi -e 's/new_stuff/old_stuff/' somefile.txt
# adding entries :)
cat somefile.txt new_records.txt > new_file.txt
As you can see, nothing in the above list is difficult, it's all in what you really need to do. However, incorporating such things into a program can sometimes be difficult. If you must use flat files, I would strongly recommend using DBD::CSV. Then, if you need to switch to a database, the conversion is trivial.
If you do decide to use a CSV database (with or without DBD::CSV), you can use my CSV Database Validation program which does a fairly rigorous job of ensuring the integrity of your data. You can use it to define basic data types, foreign key constraints and even match data against regular expressions. Create a schema, create your data and put your application through its paces. Then, run the validation script against it to ensure the integrity of your data.
I include full POD documentation in the program including a small tutorial.
Cheers,
Ovid
Join the Perlmonks Setiathome Group or just click on the the link and check out our stats. | [reply]
[d/l] |
|
|
So what's the cause of those errors
| [reply] |
|
|
| [reply] |
Re: perl cgi and flat textfile
by sparkyichi (Deacon) on Dec 07, 2001 at 05:10 UTC
|
I have found that error prone programming causes processing to be inherently error prone. My experience is that the internals of your programming (Code, Source Files, and Output) should have no errors or inconsistencies. Sometimes you might get an error from external sources such as running out of disk space or memory. Unless your program causes theses errors then there is nothing you can really do about that. It is your job as a programmer to insure that your code is as fool proof as you can make it.
I have found that if I can, I like to cerate my own Perl Database out of a text file. I guess it comes from being lazy (hey, a Perl hacker is suppose to be lazy) because connecting to a database such as Oracle, MySQL, MSAccess cause more overhead for my scripts. You would need to anticipate more things that might go wrong such as connection time-out, table unavailability and the like. If it is just a text file somewhere, then you can control the file more completely through Perl (If you need to add a table, just create a new file or what ever.) By the same token though there are times where you would need to use a managed database such as many other users and applications that need to access the data in many different ways. If you are having problems with your coding you can look at some of the many DBI modules or look in the Camel and Panther books where there are examples and explanations on how to make a working database out of a file.
Here is an example where I just write an Excel spreadsheet (Untested Sample code):
use Spreadsheet::WriteExcel;
my $julian = 'Header';
my $ttime = '1208011500';
my $workbook = Spreadsheet::WriteExcel->new("$julian.xls");
$worksheet = $workbook->addworksheet("Missing Stores $julian");
$format = $workbook->addformat();
$worksheet->set_column(0, 0, 12);
$format->set_align('center');
$format->set_bold(1);
$worksheet->write(0, 0, "Julian/Time", $format);
$worksheet->write(0, 1, "Store", $format);
$format->set_bold(0);
$worksheet->write(1, 0, "$julian/$ttime", $format);
my $i = 1;
my $n = 0;
until ( $n == $missxn ){
$worksheet->write($i, 1, "@No_file_ssc[$n]", $format);
$i++;
$n++;
}
@No_file_ssc is declaired else where in the script.
Sparky | [reply]
[d/l] |
Re: perl cgi and flat textfile
by trantor (Chaplain) on Dec 07, 2001 at 16:29 UTC
|
I agree with the previous posts, and I just wanted to point out
something more.
When dealing with CGIs and text file databases, locking is
highly recommended, because of the nature of HTTP requests themselves.
You should also consider users pressing submitting many
times or reloading the page, they both impact negatively
your flat file DB in case operations that change data are involved.
I personally like to use text files when it makes sense, e.g.
when data needs to be implicitly human readable or when
dealing with pre-existing databases and file formats.
In other situations I tend to prefer DBM files,
woinderfully supported by perl (e.g. MLDBM) and interoperable
as well, or the DBI interface,
which can operate on text files when/if needed, as already
pointed out.
-- TMTOWTDI
| [reply] |
Re: perl cgi and flat textfile
by eformat (Initiate) on Dec 07, 2001 at 20:13 UTC
|
Hi,
just as an aside, i've had very good experiences using DBI and DBD::CSV (a database driver for CSV(or tab separated) files).
Using this apporach allows you to code using familiar sql type statements, while interfacing to a flat file. If later on you want to swap to a db like MySQL no problem, simply swap the DBI driver you're using.
Mike | [reply] |
|
|
