(Solved) Search::Elasticsearch date range

bfdi533 has asked for the wisdom of the Perl Monks concerning the following question:

I have a query I need to run against ElasticSearch and am using Search::Elasticsearch module. The issue is getting the data range correctly in the query on the PERL side as I can make this work from curl without any issues. After much tinkering with format, the following no longer throws errors when executing the code but the date range does not change the number of results returned no matter what I set the date to. This leads me to believe that there is something wrong with the code.

For what it is worth, I have noticed that if I change the 'gte' to a 'lte' then I get a count of -1 showing no results are being returned. So, something is "working" if I do that ...

Any hints, changes or suggestions would be much appreciated!

#!/usr/bin/env perl

# global settings
use warnings;
use strict;
$|++;

# libraries
use Data::Dumper;
use Search::Elasticsearch;
use Try::Tiny;

# global variables
my $elk_host1 = '10.0.10.61:9200';
my $elk_host2 = '10.0.10.51:9200';
my $elk_host3 = '10.0.10.52:9200';
my $elk_host4 = '10.0.10.53:9200';
my $elk_user = 'user';
my $elk_pass = 'pass';

my $dt = `date +%F_%T`;
chomp $dt;
my $latest_dt;

print "Starting run at: ".`date`;

my $e = Search::Elasticsearch->new(
    nodes => [
        "http://$elk_user:$elk_pass\@$elk_host1" ,
        "http://$elk_user:$elk_pass\@$elk_host2" ,
        "http://$elk_user:$elk_pass\@$elk_host3" ,
        "http://$elk_user:$elk_pass\@$elk_host4" ,
    ] ,
    max_requests => 10000,
);

    my $results = $e->search (
        size => 10000,
        index => 'api-*',
        body => {
            query => {
                bool => {
                    must => {
                        term => {
                            '_type' => "alarm",
                        },
                    },
                    filter => {
                        range => {
                            '@timestamp' => {
                                gte => "2018-04-23 00:00:00",
                                format => "YYYY-MM-DD HH:mm:ss"
                            }
                        }
                    }
                }
            }
        }
    );

    print "Count: ".$#{ $results->{hits}->{hits} }."\n";
[download]

Comment on (Solved) Search::Elasticsearch date range Download Code

Replies are listed 'Best First'.
Re: Search::Elasticsearch date range by thanos1983 (Parson) on Apr 25, 2018 at 09:39 UTC
Hello bfdi533, Unfortunately I do not have anything to contribute to your question, but as I was reading your question I noticed how you collect the `date`. On the sample of code that you provide us you do not use all the following parameters anywhere `$dt` and `$latest_dt`. I assume you are using them somewhere else on your code. The way that you collect the date it might work on LinuxOS but I guess it will not work on WindowsOS (I assume as I can test it). So in order to make your code able to be executable on all OS I propose the following solution. test.pl Read more... (1025 Bytes) I am using the module Date::Manip. It is not the easiest module that you can use but the date manipulations that you can do with this module are infinite. Some examples for future study Date::Manip::Examples. Regarding the format of the date when you want to alter it, you can read more about it here: Date::Manip::Date::PRINTF_DIRECTIVES. Hope this helps for future usage. BR / Thanos Seeking for Perl wisdom...on the process of learning...not there...yet!	[reply] [d/l] [select]
Re^2: Search::Elasticsearch date range by bfdi533 (Friar) on Apr 25, 2018 at 14:21 UTC
Thank you very much for your reply. I actually use $dt and $latest_dt later in my code and did not realize I left them in the test for this post. I actually use Date::Manip as well in my code so your post/reply is very welcome. I realize that `date` is probably not the best option and does create some overhead to spawn that out. I do not need to run this on any other OS so that is not part of the consideration while developing this tool. `date` is a bad habit I picked up long ago and just need to stop copying this from code to code to code when I need the current date. I will use your example going forward as it is a much better alternative, as you have pointed out.	[reply]
Re: Search::Elasticsearch date range by bfdi533 (Friar) on Apr 25, 2018 at 18:34 UTC
Update: I found that the date format is case sensitive so what actually worked after much trial and error was the following date format string: `format => "yyyy-MM-dd HH:mm:ss"` [download] Note the small case "yyyy" and small case "dd".	[reply] [d/l]


Come for the quick hacks, stay for the epiphanies.
	PerlMonks