http://qs321.pair.com?node_id=796152


in reply to Insert BLOB into DB without using system memory, but using filehandle.

If you're dealing with files that big, why are you putting them in a database?
  1. Get an id from a database sequence,
  2. save the file using a file name based on the id, then
  3. save the id in the database.
  • Comment on Re: Insert BLOB into DB without using system memory, but using filehandle.

Replies are listed 'Best First'.
Re^2: Insert BLOB into DB without using system memory, but using filehandle.
by songahji (Friar) on Sep 21, 2009 at 12:26 UTC
    I wish I could, but the data must be in the DB. Disk space is not an issue in this project.
[reply]

      Repeat after me:

      1. I will not store files inside of the database.
      2. I will not store files inside of the database.
      3. I will not store files inside of the database.
      4. I will not store files inside of the database.
      5. I will not store files inside of the database.
      6. I will not store files inside of the database.
      7. I will not store files inside of the database.
      8. I will not store files inside of the database.

      Storing files inside of the database is just a Bad Idea. You are much better off doing something like the following:

      1. Insert a new record into the database (filename, size, source, date, etc).
      2. Get the ID of the new record.
      3. Take the md5 hash signature of the ID + some secret code.
      4. Your hash is something like 0123456789abcdef0123456789abcdef
      5. Update the database record to show "hash = '0123456789abcdef0123456789abcdef'"
      6. Make folder /var/media/01/23/45
      7. Write file contents to /var/media/01/23/45/0123456789abcdef0123456789abcdef

      NOTE: /var/media/ could be a local volume or a remote-mounted SAN. You choose.

      Now, you can partition the files across multiple servers. You can store the files on a SAN and have (many) dummy file servers in front of it.

      This is (basically) how wikimedia does it. And just look at how many files they have. Storing the files inside of the database is just Not How It's Done.

[reply]
[d/l]
[select]
        Your idea is better, I recon. I wish I am the boss, but gotta do what the boss want. Cheers!
[reply]
        Wow. Such arrogance.
[reply]

In Section Seekers of Perl Wisdom