Day: February 9, 2024

Posted on

KoreaDaily Yellowpage Mechanize Perl Code

using below code based extracted URL, use below Perl Mechanize to download all pages of each business:

#!/usr/bin/perl

use strict;
use warnings;
use Encode;  # To handle encoding issues
use WWW::Mechanize;

# Read URLs from the text file
my $filename = 'urls.txt';

# Create a new WWW::Mechanize object
my $mech = WWW::Mechanize->new();

# Open the text file
open(my $fh, '<', $filename) or die "Could not open file '$filename' $!";

# Initialize page number
my $page_num = 1;

# Loop through each URL in the file
while (my $url = <$fh>) {
    chomp $url;  # Remove newline character

    # Visit the URL
    $mech->get($url);

    # Get the content of the current page
    my $content = $mech->content();

    # Save the content to a file with EUC-KR encoding
    my $filename = sprintf("%d.html", $page_num);
    open(my $fh_out, '>:encoding(EUC-KR)', $filename) or die "Could not open file '$filename' for writing: $!";
    print $fh_out $content;
    close $fh_out;

    # Find pagination links and click on them to navigate through pages
    my @pagination_links = $mech->find_all_links(url_regex => qr/page=/i);
    for my $link (@pagination_links) {
        $mech->get($link->url);
        my $content = $mech->content();
        $page_num++;
        my $filename = sprintf("%d.html", $page_num);
        open(my $fh_out, '>:encoding(EUC-KR)', $filename) or die "Could not open file '$filename' for writing: $!";
        print $fh_out $content;
        close $fh_out;
    }

    # Increment page number
    $page_num++;
}

# Close the file handle
close($fh);
Posted on

KoreaDaily Yellowpage – each business URL

download as txt file using Firefox and use below code to extract each business URL:

#!/usr/bin/perl

use strict;
use warnings;

# Define the output file name
my $output_file = 'KoreaDaily.txt';

# Open the output file for writing
open(my $fh_out, '>', $output_file) or die "Could not open file '$output_file' for writing: $!";

# Loop through files Atlanta_01.txt to Atlanta_13.txt
for my $file_number (1..14) {
    my $input_file = "Atlanta_" . sprintf("%02d", $file_number) . ".txt";
    
    # Open the current input file for reading
    open(my $fh_in, '<', $input_file) or die "Could not open file '$input_file': $!";

    # Iterate through each line of the current input file
    while (my $line = <$fh_in>) {
        # Check if the line contains 'cat_code='
        if ($line =~ /cat_code=/) {
            # Write the line to the output file
            print $fh_out $line;
        }
    }

    # Close the current input file handle
    close($fh_in);
}

# Close the output file handle
close($fh_out);

print "Extraction completed. Results saved in '$output_file'.\n";