Comment utiliser LWP :: UserAgent de Perl pour récupérer la même URL avec différentes chaînes de requête?

Question

J'ai un courant LWP :: UserAgent qui doit être appliqué sur l'URL suivante:

http://dms-schule.bildung.hessen.de/suchen/suche_schul_db.html?show_school=5503 

Cela va avec beaucoup de cibles similaires voir les fins suivantes:

html?show_school=5503 
html?show_school=9002 
html?show_school=5512 

Je veux le faire avec utiliser LWP :: UserAgent:

for my $i (0..10000) 

{ $ua->get(' [here the URL should be applied] ', id => 21, extern_uid => $i); 
# process reply } 

Dans tous les cas, en utilisant une boucle comme celui-ci pour ce genre de travail est une façon de le faire. Je suppose que l'API de LWP ne vise pas à remplacer la fonctionnalité du noyau Perl, et je peux utiliser des boucles Perl pour interroger plusieurs URL.

Le code qui ne fonctionne pas parce que la boucle doit être appliquée:

#use strict; 

use DBI; 
use LWP::UserAgent; 
use HTTP::Request::Common; 
use HTML::TreeBuilder::XPath; 

# first get a list of all schools 
my ($url = '[here the url should be applied] =',id); 

for my $id (0..10000) { 
    $ua->get(' [here the url should be applied ] ', id => 21, extern_uid => $i); 
    # process reply 
} 

#my $request = POST $url, 
#     [ 
#   Schulsuche=> "Ergebnisse anzeigen", 
#   order => "schule_ort", 
#   schulname => undef, 
#   schulort => undef, 
#   typid => "11", 
#   verbinder => "AND" 
#     ]; 

my $ua = LWP::UserAgent->new; 
print "getting all schools - this could take some time\n"; 
my $response = $ua->request($request); 

# extract the ids 
my @ids = $response->content =~ /getSchoolDetail\((\d+)/gs; 
print "found " . scalar @ids . " schools\n"; 

# for this demo we only do the first 5 
my @ids_to_do = @ids[0..4]; 

# use your own user and password 
my $dbh = DBI->connect("DBI:mysql:database=schulen", "user", "pass", { AutoCommit => 0 }) or die $!; 

my $sth = $dbh->prepare(<<sqlend); 
    insert into schulen (name , plz , ort, strasse , tel, fax , mail, quelle , original_id) 
       values (?, ?, ?, ?, ?, ?, ?, ?, ?) 
sqlend 

# now loop over ids 
for my $id (@ids_to_do) { 

    # get detail information for id 
    my $res = $ua->get("[url]=> &gid=$id"); 

    # parse the response 
    my $tree = HTML::TreeBuilder::XPath->new; 
    $tree->parse($res->content); 

    my $xpath = q|//div[@id='MCinhview']//div[@class='contentitem']//table|; 
    my ($adress_table, $tel_table) = $tree->findnodes($xpath); 

    my ($adr) = $adress_table->find("td"); 
    my ($name, $city, $street) = map { s/^\s*//; s/\s*$//; $_ } ($adr->content_list)[2,4,6]; 

    my($plz, $ort) = $city =~ /^(\d+)\s*(.*)/; 
    my ($tel, $fax, $mail) = map { s/^\s*//; s/\s*$//; $_ } map { ($_->content_list)[1] } $tel_table->find("td"); 

    $sth->execute($name, $plz, $ort, $street, $tel, $fax, $mail, "SA", $id); 
    $dbh->commit; 

    $tree->delete; 

    print "$name done\n"; 
} 

---

mise à jour le dimanche 25 octobre: ​​ J'ai appliqué les conseils de OmnipotentEntity.

#!/usr/bin/perl -W 

use strict; 
use warnings;   # give out some warnings if something does not run well 
use diagnostics;  # tell me when something is wrong 
use DBI; 
use LWP::UserAgent; 
use HTTP::Request::Common; 
use HTML::TreeBuilder::XPath; 

# first get a list of all schools 

my $ua = LWP::UserAgent->new; 

$ua->agent("Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7"); 

#pretending to be firefox on linux. 

for my $i (0..10000) { 
    my $request = HTTP::Request->new(GET => sprintf(" here to put the URL into =%d", $i)); 
    $request->header('Accept' => 'text/html'); 
    my $response = $ua->request($request); 
    if ($response->is_success) { 
    $pagecontent = $response -> content; 
    } 
# now we can do whatever with the $pagecontent 

} 
my $request = POST $url, 
[ 
      order => "schule_ort", 
      schulname => undef, 
      Basisdaten => undef,   
      Profil => undef, 
      Schulort => undef, 
      typid => "11", 
      Fax => 
      Homepage => undef, 
      verbinder => "AND" 

]; 

print "getting all schools - this could take some time\n"; 
my $response = $ua->request($request); 

# extract the ids 
my @ids = $response->content =~ /getSchoolDetail\((\d+)/gs; 
print "found " . scalar @ids . " schools\n"; 

# for this demo we only do the first 5 
my @ids_to_do = @ids[0..4]; 

# use your own user and password 
my $dbh = DBI->connect("DBI:mysql:database=schulen", "user", "pass", { AutoCommit => 0 }) or die $!; 

my $sth = $dbh->prepare(<<sqlend); 
    insert into schulen (name , plz , ort, strasse , tel, fax , mail, quelle , original_id) 
       values (?, ?, ?, ?, ?, ?, ?, ?, ?) 
sqlend 

# now loop over ids 
for my $id (@ids_to_do) { 

    # get detail information for id 
    my $res = $ua->get(" here to put the URL into => &gid=$id"); 

    # parse the response 
    my $tree = HTML::TreeBuilder::XPath->new; 
    $tree->parse($res->content); 

    my $xpath = q|//div[@id='MCinhview']//div[@class='floatbox']//table|; 
    my ($adress_table, $tel_table) = $tree->findnodes($xpath); 

    my ($adr) = $adress_table->find("td"); 
    my ($name, $city, $street) = map { s/^\s*//; s/\s*$//; $_ } ($adr->content_list)[2,4,6]; 

    my($plz, $ort) = $city =~ /^(\d+)\s*(.*)/; 
    my ($tel, $fax, $mail) = map { s/^\s*//; s/\s*$//; $_ } map { ($_->content_list)[1] } $tel_table->find("td"); 

    $sth->execute($name, $plz, $ort, $street, $tel, $fax, $mail, "SA", $id); 
    $dbh->commit; 

    $tree->delete; 

    print "$name done\n"; 
} 

Je veux faire une boucle sur les résultats et donc j'ai essayé d'appliquer les URL correspondantes mais je me suis un tas d'erreurs:

 
suse-linux:/usr/perl # perl perl_mecha_example_two.pl 
Global symbol "$pagecontent" requires explicit package name at perl_mecha_example_two.pl line 24. 
Global symbol "$url" requires explicit package name at perl_mecha_example_two.pl line 29. 
Execution of perl_mecha_example_two.pl aborted due to compilation errors (#1) 
    (F) You've said "use strict" or "use strict vars", which indicates 
    that all variables must either be lexically scoped (using "my" or "state"), 
    declared beforehand using "our", or explicitly qualified to say 
    which package the global variable is in (using "::"). 

Uncaught exception from user code: 
Global symbol "$pagecontent" requires explicit package name at perl_mecha_example_two.pl line 24. 
Global symbol "$url" requires explicit package name at perl_mecha_example_two.pl line 29. 
Execution of perl_mecha_example_two.pl aborted due to compilation errors. 
at perl_mecha_example_two.pl line 86 

Maintenant, la partie de débogage. Qu'est-ce que je change? Comment appliquer les URL de la bonne manière? Lorsque j'utilise strict je ne suis pas autorisé à utiliser une variable avant de le déclarer. La solution habituelle consiste à ajouter un préfixe my, par ex. my $url et my $pagecontent sur la première apparition de celui-ci.

Answer 1

Il est aussi simple que:

#!/usr/bin/perl -W 

use LWP::UserAgent; 

my $ua = LWP::UserAgent->new; 
$ua->agent("Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7"); #pretending to be firefox on linux. 
for my $i (0..10000) { 
    my $req = HTTP::Request->new(GET => sprintf("http://path/to/url?=%d", $i)); 
    $req->header('Accept' => 'text/html'); 
    my $res = $ua->request($req); 
    if ($res->is_success) { 
    $pagecontent = $res -> content; 
    } 
# Do whatever with the $pagecontent 
} 

Ceci suppose que vous voulez récupérer toutes les 10000 pages. Si vous voulez seulement récupérer des éléments particuliers, vous devriez essayer de lancer ces nombres dans un tableau, et ensuite avoir pour marcher ce tableau, plutôt que 1..10000