http://qs321.pair.com?node_id=11128318

throwaway has asked for the wisdom of the Perl Monks concerning the following question:

using LWP user agent and gettting error 1010 for every website i try besides setup api json endpoints, this is the error seen in terminal

HTTP/1.1 403 Forbidden Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate +, post-check=0, pre-check=0 Connection: close Date: Sat, 13 Feb 2021 02:04:56 GMT Server: cloudflare Content-Length: 16 Content-Type: text/plain; charset=UTF-8 Expires: Thu, 01 Jan 1970 00:00:01 GMT CF-RAY: 620afb475bb5f242-ORD Cf-Request-Id: 083abb60980000f242471b1000000001 Client-Date: Sat, 13 Feb 2021 02:04:56 GMT Client-Peer: 104.19.128.108:443 Client-Response-Num: 1 Client-SSL-Cert-Issuer: /C=US/O=Cloudflare, Inc./CN=Cloudflare Inc ECC + CA-3 Client-SSL-Cert-Subject: /C=US/ST=CA/L=San Francisco/O=Cloudflare, Inc +./CN=sni.cloudflaressl.com Client-SSL-Cipher: TLS_AES_256_GCM_SHA384 Client-SSL-Socket-Class: IO::Socket::SSL Client-SSL-Version: TLSv1_3 Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.c +om/cdn-cgi/beacon/expect-ct" Set-Cookie: __cfduid=d8ad08b4932ed14004949bb070cb4f43f1613181896; expi +res=Mon, 15-Mar-21 02:04:56 GMT; path=/; domain=.4chan.org; HttpOnly; + SameSite=Lax; Secure X-Frame-Options: SAMEORIGIN error code: 1010

This is the exact code im using except url

use LWP::UserAgent; # specify a CA path my $ua = LWP::UserAgent->new(); my $response = HTTP::Request->new(GET => 'https://example.com/'); my $res = $ua->request($response)->as_string; print $res

Replies are listed 'Best First'.
Re: LWP::UserAgent throwing 1010 error
by jcb (Parson) on Feb 13, 2021 at 03:02 UTC

    You are using a website "Proudly Broken by CloudFlare" and the error you are getting is intentional.

    In fact, details in your post indicate that the site in question is 4chan. Why are you pointing LWP::UserAgent at 4chan?

      "Why are you pointing LWP::UserAgent at 4chan?"

      For the lulz I'm guessing...

Re: LWP::UserAgent throwing 1010 error
by throwaway (Novice) on Feb 13, 2021 at 02:16 UTC
    seems like something to do with cloudfare? maybe some authorization that im not configuring?
Re: LWP::UserAgent throwing 1010 error
by throwaway (Novice) on Feb 13, 2021 at 14:09 UTC
    why not its simple html to use when im learning lol but i was using wrong urls!!!! i guess the endpoints were changing from what I would use in the browser to what was actually having a connection in the end :/
Re: LWP::UserAgent throwing 1010 error
by throwaway (Novice) on Feb 13, 2021 at 14:55 UTC
    any advice on matching nested structures like html would also be appreciated
        thank you those are great, I think I will try MOJO

      "any advice on matching nested structures like html would also be appreciated "

      Here you go: Never send a regexp to do a parser's work.

      Hope this helps!


      The way forward always starts with a minimal test.
        ahhh true it does, thanks for that
Re: LWP::UserAgent throwing 1010 error
by throwaway (Novice) on Feb 13, 2021 at 14:20 UTC
    4chan was becoming 4channel in this case lmao