Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re^2: DBD::SQLite - Can't Connect to file - Solved

by wjw (Priest)
on Jan 17, 2019 at 18:37 UTC ( [id://1228697]=note: print w/replies, xml ) Need Help??


in reply to Re: DBD::SQLite - Can't Connect to file - Solved
in thread DBD::SQLite - Can't Connect to file - Solved

Thanks so much for taking the time to review my code and offer up excellent suggestions. I apologize for not responding sooner. There have been some health issues that have kept me away for a while. I will be making use of these suggestions, of that you can be certain!

I do want to explain (excuse) the bit about connecting and disconnecting from the DB each time. I too generally leave the connection open and reuse it. My concern has been power usage and power interruption. This system is powered by a solar panel/battery setup in an area where there can be some fairly extreme weather and intermittent supply of light. I have some protection on the Pi which shuts it down gracefully when voltage drops low enough in the battery. And yet I have had some instances where power suddenly drops out. My concern was that the DB file might get corrupted were it being accessed at the point at which power fails. Thus, I open and close that DB file the way that I do. I may be wasting my time on that, but it seemed reasonable to me.

Power consumption is pretty high on a RPi3, and there is no good on-board way of reducing that. There are however some add-ons (which I have yet to order) which will act as 'wake-on' devices. I may use one of these to address battery life in the field. Those too, literally shut down the Pi and restart it on some detected event. I figured that when I implement that, the DB being shut down is one less worry.

My long term intent is to use a MySQL (Maria) DB, instead of the SQLite DB. But that won't happen until I get LoRa enabled Arduinos doing the data gathering and communication. At that point, the Pi will be at a location where standard power is available and consumption will not be a concern. Going to be a while though... Lots of learning to do first.

At any rate, I wanted you to know that your efforts are greatly appreciated. This to me is what Perl Monks is about. I never fail to learn something here...

...the majority is always wrong, and always the last to know about it...

A solution is nothing more than a clearly stated problem...

  • Comment on Re^2: DBD::SQLite - Can't Connect to file - Solved

Replies are listed 'Best First'.
Re^3: DBD::SQLite - Can't Connect to file - Solved
by afoken (Chancellor) on Jan 18, 2019 at 22:47 UTC
    This system is powered by a solar panel/battery setup in an area where there can be some fairly extreme weather and intermittent supply of light. I have some protection on the Pi which shuts it down gracefully when voltage drops low enough in the battery. And yet I have had some instances where power suddenly drops out.

    That sounds like an electrical problem. Perhaps simply a loose connection or corrosion on contacts? Extreme weather really sounds like corrosion.

    Or maybe your battery is so worn out that its output voltage suddenly drops rapidly, much faster than you expect. What kind of battery do you use?

    • Li-Ion / Li-Polymer self-destruct within three to five years, no matter what you do. And they absolutely do not like being over-charged or deeply discharged. So lithium cells often come with build-in protection circuits that simply cut power if you abuse the cell too much.
    • Lead-acid batteries release hydrogen and oxygen while charging, especially when over-charging (more than 2.4 V/cell), so you need to refill them with destillated water from time to time. They don't like deep discharging (below 1.8 V/cell), but otherwise, they are robust and reliable.
    • Maintenance-free lead-acid batteries aren't. Once they have released too much hydrogen and oxygen, they are broken, as you can't refill water. The gel-based ones have the advantage that you can use them in any orientation.
    • NiCd is old technology, they don't like beging charged before they are completely discharged (memory effect). In a solar panel + battery setup, a NiCd battery is clearly the wrong choice.
    • NiMH is an improvement over NiCd, has nearly no memory effect, but can't hold has much energy as Li-Ion / Li-Polymer cells and isn't as robust as lead-acid batteries.
    • LiFePO4 is more rubust than Li-Ion/Li-Polymer, does not self-destruct as fast as Li-Ion/Li-Polymer, but has a slightly lower voltage and less nominal capacity. Still, expect to find protection circuits on LiFePO4 cells.

    My concern was that the DB file might get corrupted were it being accessed at the point at which power fails. Thus, I open and close that DB file the way that I do. I may be wasting my time on that, but it seemed reasonable to me.

    Sounds sane. There are other options, such as using a robust filesystem that can handle unexpected blackouts and guarantee that any change to a file is done atomically.

    For another option, that can be used in combination, see below.


    Power consumption is pretty high on a RPi3, and there is no good on-board way of reducing that. There are however some add-ons (which I have yet to order) which will act as 'wake-on' devices. I may use one of these to address battery life in the field. Those too, literally shut down the Pi and restart it on some detected event. I figured that when I implement that, the DB being shut down is one less worry.

    The Raspi3 is documented to need 5 V at a maximum of 2 A. One problem of the Raspi in general is the Micro-USB-Connector used for power supply. It was originally spec'd for 0.5 A, but the connector can transport 2.5 A to 3 A before burning. The contacts are tiny, and the wires in many USB cables are very thin, so you loose significant voltages on the cable.

    The Raspberry Pi foundation recommends "a good-quality power supply that can supply at least 2A at 5V for the Model 3B". Their official power supply delivers up to 2.5 A at 5.1 V, allowing a drop of 0.1 V at cable and connectors.

    A better way to supply the Raspi is the GPIO connector (at the 5V lines), using short, thick wires or a shield with short, wide traces from the voltage regulator to the pins.

    If unexpected blackouts may happen in the solar panel+battery system, consider using a local buffer battery connected directly to the Raspi. There are some ready-to-use solutions, see for example this article from the german c't magazine. You probably don't need a huge battery, it just has to reliably deliver sufficient power for a clean shutdown. Ideally, the battery can power the raspi for a few minutes so a short blackout does not cause an immediate shutdown. The raspi should be able to detect the remaining battery charge, so it can shut down before the battery is completely discharged.


    My long term intent is to use a MySQL (Maria) DB, instead of the SQLite DB.

    I would recommend using PostgreSQL. Here is why: Re^2: What is the best Perl library to use to connect and access MSSQL2008 Database from winXP PC?, Re^2: Perl and Database, Re^2: What is your favourite Linux or cross-platform database?. PostgreSQL is quite easy to set up and run, and it comes with excellent documentation. You may also want to install PgAdmin III, perhaps on a different computer.

    Alexander

    --
    Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
Re^3: DBD::SQLite - Can't Connect to file - Solved
by roboticus (Chancellor) on Jan 18, 2019 at 03:05 UTC

    wjw:

    No worries on response time--real life often happens to get in the way of things.

    It sounds like an interesting project. I expect you already know, but just in case you don't, you might want to look over some of stevieb's stuff--he does a bunch of RPi/Perl stuff, including a book and some cpan modules.

    ...roboticus

    When your only tool is a hammer, all problems look like your thumb.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1228697]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others about the Monastery: (5)
As of 2024-04-19 16:12 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found