This article is about How to do simple ruby web scraping by processing CSV.
In this article, we will create a Ruby on Rails application to scrap the link uploaded from a CSV file and find the occurrence of the link on a particular page.
In the below Ruby on Rails application development, the user needs to pass a CSV file and list of the user’s email to whom the parsed CSV will be sent.
In the CSV file, there will be 2 columns
Let’s start creating a Rails Application
$ rails new scrape_csv_data
$ cd scrape_csv_data
Then, we will generate an Upload CSV module. Run the below command.
$ rails g scaffold UploadCsv generated_csv:string csv_file:string
This will create all the required models, controllers, and migrations for csv_file. Run the migration using the below command.
$ rails db:migrate
gem 'carrierwave', '~> 2.0'
$ bundle install
$ rails generate uploader Avatar
class UploadCsv < ApplicationRecord
mount_uploader :csv_file, AvatarUploader
Update the routes.
Then, we will start the server and check if the application is working successfully or not.
$ rails s
$ rails generate job genrate_csv
Now we will send the generated file through email by using the below instructions.
After uploading the file check the scrap generated file will be updated. You can check the generated file in /scrape_data/public/result_data.csv
First, we will generate the mailer by using the below command.
$ rails generate mailer NotificationMailer
Add this code inside the notification mailer.
Also, we need to add mail configuration inside config/environments/development.rb or production.rb.
Also, we need to update the view also app/views/notification_mailer/send_csv.html.erb
Source Link: https://www.botreetechnologies.com/blog/web-scraping-with-ruby-by-processing-csv/