Python | TekCookie

Web Scraping with Python

jeffythampi — Tue, 04 Aug 2020 06:37:27 +0000

Web Scraping is the method used for extracting or harvesting data from websites. The extracted data is then processed and stored as a more structed form.

Everyone of us might have copied some information from websites for our requirements. Web scraping also extracts information from the websites, but in huge volume by means of any script of special software’s.

When we do a web search, search engines present to us tonnes of information from millions of websites. How do they do that? It is done by crawling through the websites and fetching the information to build their database.

This article is about how to scrape information from a demo web page (a simple one) using python.

About legality, please do your research !!!

Note: This is article only for the purpose of knowledge sharing.

The demo web page

HTML code of website







	
		Name
		SerialNo
		Price
	
	
		Butter
		22315
		12
	
	
		Gum
		11452
		5
	
	
		Milk
		55462
		23
	
	
		Sugar
		55411
		18

Name	SerialNo	Price
Butter	22315	12
Gum	11452	5
Milk	55462	23
Sugar	55411	18

Python code to scrape the information from website

import requests
import bs4
import csv


#class definition
#This class objects hold the data of each item
class Data_Class:
  col_data_1 = None
  col_data_2 = None
  col_data_3 = None

  def __init__(self, col_data_1, col_data_2, col_data_3):
    self.col_data_1 = col_data_1
    self.col_data_2 = col_data_2
    self.col_data_3 = col_data_3

  def __iter__(self):
    return iter([self.col_data_1, self.col_data_2, self.col_data_3])


#Functions
#This function writes the list of objects to a csv file
def write_csv(data_list, file_path):
  with open(file_path, 'w', newline='') as file:
    writer = csv.writer(file)
    writer.writerows(data_list)


#This function will fetch the website data and pass the deta to write_csv function
def fetch_webdata(filepath, weburl):
  data_list = list()
  row_count = 0
  #get the website data and parse using beautifulsoup
  response = requests.get(weburl)
  soup = bs4.BeautifulSoup(response.content, 'html.parser')
  #find all the table rows and loop through the items
  allrows = soup.find_all('tr')
  for row in allrows:
    #Fetching the heading
    if (row_count == 0):
      row_count += 1
      allheadings = row.find_all('th')
      head_count = 0
      head_name_1 = ""
      head_name_2 = ""
      head_name_3 = ""
      for heading in allheadings:
        if (head_count == 0):
          head_name_1 = heading.get_text()
        elif (head_count == 1):
          head_name_2 = heading.get_text()
        else:
          head_name_3 = heading.get_text()
        head_count = head_count + 1
      data_class_obj = Data_Class(head_name_1, head_name_2, head_name_3)
      data_list.append(data_class_obj)
      continue
    #Fetching the rows
    #in each row, get the columm and loop through values
    allcols = row.find_all('td')
    col_count = 0
    col_data_1 = ""
    col_data_2 = 0
    col_data_3 = 0
    #loop through the values in the sequence as they appear in the html table
    for col in allcols:
      if (col_count == 0):
        col_data_1 = col.get_text()
      elif (col_count == 1):
        col_data_2 = col.get_text()
      else:
        col_data_3 = col.get_text()
      col_count = col_count + 1
    #assign the vales to object of the declared class "Data_Class"
    data_class_obj = Data_Class(col_data_1, col_data_2, col_data_3)
    #append the object to the list
    data_list.append(data_class_obj)
  #calling the write csv function to write the data to a csv file
  write_csv(data_list, filepath) 


filepath = r'C:\Users\weapon-x\Python\File.csv'
weburl = 'https://tekcookie.com/samplepage.html'
fetch_webdata(filepath, weburl)

The above code will save the data as a csv file

When the website has multiple tables and complex contents, the code has to be modified to access the required table by the class id, table names, etc.

Hope this article is informative and thank you for reading.

Reverse a String in Python

jeffythampi — Sat, 25 Jul 2020 09:46:21 +0000

In Python, string object is considered as a sequence of characters represented by double quotes (“”) or single quotes (”).

To see how to reverse the string in PowerShell, have a look at https://tekcookie.com/reverse-a-string-in-powershell/

Accessing characters through array index

s = "tekcookie.com"

#To get the first character
print("First Character: ", s[0])

#To get the last character
print("Last Character: ", s[-1])

Result:

Display full string

s = "tekcookie.com"

#To display full string
print("String is: ", s)

#Iterating through all indexes
print("String is: ", s[0:len(s):1])

#In short form
print("String is: ", s[::1])

#Skipping alternate characters
print("Skipping alternate characters in String: ", s[::2])

Result:

Reversing the String

Reverse string by iterating through array

s = "tekcookie.com"

#Iterating all indexes - 
print("Reversed string:", s[-1:-len(s)-1:-1])

#In Short
print("Reversed string:", s[::-1])

Result

Reverse string using for loop

s = "tekcookie.com"

strn = ""
for i in s:
  strn = i + strn
print("Reversed String:", strn)

Result:

Reverse string using reversed method

s = "tekcookie.com"
print("Reversed string:", ''.join(reversed(s)))

Result:

If you know of other ways to reverse the string in Python, comment below with the code !!!

Thank you for reading the article.

Edit Multiple CSV Files using Python

jeffythampi — Thu, 30 Apr 2020 19:30:00 +0000

In my previous post, we have seen how to read CSV files https://tekcookie.com/read-csv-file-using-python/ and how to write data to text files https://tekcookie.com/write-to-a-text-file-using-python/

This article is about how to modify or edit multiple CSV files using Python.

Suppose we have the daily sale data of apples as shown below. Our task is to add one column to the right and calculate the total sale in each week.

Source Data

Required Result

Source content in raw csv

Weeks,Monday,Tuesday,Wednesday,Thursday,Friday
Week-1,20,41,22,65,42
Week-2,23,23,56,44,85
Week-3,54,54,68,56,44
Week-4,12,89,32,23,65

Logic

Open the CSV file and read the data
Iterate through each row and calculate the sum of values for each week.
Append the calculated data to the row
Overwrite to the existing file

Python code to edit CSV file

import csv


# function to write the list to csv file
def write_csv(data_list, file_path):
    with open(file_path, 'w', newline='') as file:
        writer = csv.writer(file)
        writer.writerows(data_list)


# function to read CSV to list
def read_csv(file_path):
    csv_data_list = []
    source_file = open(file_path)
    csv_data = csv.reader(source_file)
    for data in csv_data:
        # converting the data to list
        csv_data_list += [data]
    source_file.close()
    return csv_data_list


# main function
def edit_csv_main(file_path):
    csv_data = read_csv(file_path)
    final_list = []
    for row_num in range(0, len(csv_data)):
        # assigning the list to a variable for readability
        data = csv_data[row_num]
        total = 0
        # adding heading to the first row
        if row_num == 0:
            data.append("Total")
        else:
            # total of each item in the row except the first item as it is a text
            for numbers in data[1:len(data)]:
                total += int(numbers)
            # append the result to end of the list
            data.append(total)
        # append the result to final list
        final_list += [data]

    # calling csv write function
    write_csv(final_list, file_path)

# calling the main function with path to the source file
edit_csv_main(r'D:\BLOG\Python\WriteCSV\File-17-04-2020.csv')

Running the above code will give us the required result.

If there are multiple files in which the data has to be modified, a small change to the above code can address it.

Python code to edit multiple CSV file

Suppose all the CSV files are kept at “D:\BLOG\Python\WriteCSV\” and the file names are File-.csv

The way is to get path of the file of type CSV having names starting with File* and pass it to the code (Line# 53 in the code below).

We use glob function to find the file names in the folder (Line# 27 in the code below).

import csv
import glob


# function to write the list to csv file
def write_csv(data_list, file_path):
    with open(file_path, 'w', newline='') as file:
        writer = csv.writer(file)
        writer.writerows(data_list)
        #for data in data_list:
        #    writer.writerow(data)

# function to read CSV to list
def read_csv(file_path):
    csv_data_list = []
    source_file = open(file_path)
    csv_data = csv.reader(source_file)
    for data in csv_data:
        # converting the data to list
        csv_data_list += [data]
    source_file.close()
    return csv_data_list


# main function
def edit_csv_main(file_path):
    file_names = glob.glob(file_path)
    for file_path in file_names:
        csv_data = read_csv(file_path)
        final_list = []
        for row_num in range(0, len(csv_data)):
            # assigning the list to a variable for readability
            data = csv_data[row_num]
            total = 0
            # adding heading to the first row
            if row_num == 0:
                data.append("Total")
            else:
                # total of each item in the row except the first item as it is a text
                for numbers in data[1:len(data)]:
                    total += int(numbers)
                # append the result to end of the list
                data.append(total)
            # append the result to final list
            final_list += [data]

        # calling csv write function
        write_csv(final_list, file_path)


# calling the main function
# here the 
edit_csv_main(r'D:\BLOG\Python\WriteCSV\File*.csv')

Running the above code will edit all the CSV files in the mentioned location having their names starting with File*

Thank you for reading my post. Hope this is informative for you.

Write to a text file using Python

jeffythampi — Sun, 19 Apr 2020 16:58:33 +0000

This article will describe how to write content to a text file using python

Write one line to text file

Set the argument to ‘w’ to write to a file. If the file does not exists, new file is created

file = open("demo.txt", 'w')
file.write("this is first line")
file.close()

Output:

Write two lines to text file

file = open("demo.txt", 'w')
file.write("this is first line")
file.write("\n")
file.write("this is second line")
file.close()

Output:

Append data to a text file

Set the argument to ‘a’ to append to a file. If the file does not exists, new file is created

"r" - Default mode, open the file in read mode
"x" - Create a file, returns an error if the file exist
"a" - Append to the file, will create a file if the specified file does not exist
"w" - Write to the file, will create a file if the specified file does not exist

file = open("demo.txt", 'a')
file.write("\n")
file.write("this is third line")
file.write("\n")
file.write("this is fourth line")
file.close()

Output:

Write a list of string to text file

string = ["first line", "\n", "second line", "\n", "third line"]
file = open("demo.txt", 'w')
file.writelines(string)
file.close()

Note: file.write() function will write string to text file and file.writelines() will write a list of strings to text file

Output:

Write from 1 to 10 to text file

file = open("demo.txt", 'w')
for num in range(1,10):
    file.write(str(num) + "\n")
file.close()

Output:

Copy content from one file to other

file = open("demo.txt")
with open("dest_demo.txt",'w') as dest_demo:
    dest_demo.writelines(file.readlines())
file.close()

Copy content from demo.txt to dest_demo.txt

Output:

Thank you for reading this article. Hope this is helpful to you.

Read CSV file using Python

jeffythampi — Mon, 30 Mar 2020 11:52:32 +0000

In this article, we will see how to read information from a CSV file and modify the imported data using Python. CSV package is used to read csv file information.

Guys, click me to have a look at my article describing how to backup up network devices using Python.

Tip: I’m using PyCharm community edition IDE for writing Python scripts.

Following csv file is saved at C:\SW\data.csv

The first row in the csv represent the heading

Name,Price,qty
Mango,20.3,4
Orange,23.1,6
Apple,56.3,9

Read and print CSV file

import csv

sourceFile = open("C:/SW/data.csv")
csvData = csv.reader(sourceFile)

print(list(csvData))
sourceFile.close()

Output:
[['Name', 'Price', 'qty'], ['Mango', '20.3', '4'], ['Orange', '23.1', '6'], ['Apple', '56.3', '9']]

Looping through CSV file

import csv 

sourceFile = open("C:/SW/data.csv")
csvData = csv.reader(sourceFile)
for rows in csvData:
    print(rows)

sourceFile.close()

Output: 
['Name', 'Price', 'qty']
['Mango', '20.3', '4']
['Orange', '23.1', '6']
['Apple', '56.3', '9']

Print data without heading

import csv

sourceFile = open("C:/SW/data.csv")
csvData = csv.reader(sourceFile)
for rows in csvData:
#first line represent heading
    if csvData.line_num == 1:
        continue
    print(rows)

sourceFile.close()

Output:
['Mango', '20.3', '4']
['Orange', '23.1', '6']
['Apple', '56.3', '9']

Print the items from which the price is greater than 22

import csv

sourceFile = open("C:/SW/data.csv")
csvData = csv.reader(sourceFile)

#for serial number while displaying the data 
r_count = 1

#each row item is assigned to variables name, price and quantity 
for name, price, quantity in csvData:
    if csvData.line_num == 1:
        continue
    if float(price) > 22:
        print("{}. Price of {} is {}".format(r_count, name, price))
        r_count += 1

sourceFile.close()

Output: 
Price of Orange is 23.1
Price of Apple is 56.3

Adding column to the list of item

inport csv

sourceFile = open("C:/SW/data.csv")
csvData = csv.reader(sourceFile)

#adding extra column to the list
newlist = [x + [0] for x in csvData]
print(newlist) #Output - 1

#to get the data without heading (skipping the first row)
newlist = newlist[1:len(newlist)]
print(newlist)  #Output - 2

#below code will calculate the total amount to newely added column
# newColumn = Price * Quantity
r_count = 1
newlist1 = []
for row in newlist:
    row[3] = float(row[1]) * float(row[2])
    newlist1.append(row)

print(newlist1)  #Output - 3

sourceFile.close()

Output - 1:  
[['Name', 'Price', 'qty', 0], ['Mango', '20.3', '4', 0], ['Orange', '23.1', '6', 0], ['Apple', '56.3', '9', 0]] 

Output - 2:  
[['Mango', '20.3', '4', 0], ['Orange', '23.1', '6', 0], ['Apple', '56.3', '9', 0]]   

Output - 3:  
[['Mango', '20.3', '4', 81.2], ['Orange', '23.1', '6', 138.6], ['Apple', '56.3', '9', 506.7]]

Thank you for reading my post. Hope this is helpful to you.

Automate Cisco switch backup using Python

jeffythampi — Sun, 30 Jun 2019 13:15:54 +0000

This python script reads the device information from a csv file and fetches its configuration via telnet session and save to specified location.

CSV Template:

deviceIp,userName,password 
192.168.1.1,admin,secretpwd 
192.168.1.2,admin,secretpwd2

The script initiates telnet session with the device along with userId and password to login into enable mode and executes “show run” command to fetch the configuration

Python code to fetch switch configuration:

import telnetlib as tel
import csv
import datetime
import os


#Declaration
#path of the csv file
#deviceFilePath = "D:/Script/DeviceIp.csv"
#path to save the backup files
#directoryLocation = "D:/Script/"


#Methods
###This method initiate telnet connection to devices
def Initiate_Telnet(indexCount, hostIP, userId, passwd, directoryLocation):
    commands_ShowConfig = "sh run"
    commands_terminalLength = "terminal length 0"
    response = os.system("ping -n 1 -w 700 " + hostIP)
    if response == 0:
        try:
            #connecting to host device
            telConnection = tel.Telnet(hostIP, timeout=2)

            #sending username value and password value
            uName = telConnection.read_until('Username:'.encode('utf-8'), timeout=5)
            telConnection.write(userId.encode('utf-8') + b"\n")
            uPass = telConnection.read_until('Password:'.encode('utf-8'), timeout=5)
            telConnection.write(passwd.encode('utf-8') + b"\n")

            #checking for "#"
            telConnection.write(b"\n")
            mode = telConnection.read_until('#'.encode('utf-8'), timeout=5)
            #x = b' \r\nBZPDAMG0204#'
            mode = str(mode.decode('utf-8'))

            if(mode[-1] == "#"):
                #sending terminal length command to get full configuration
                telConnection.write(commands_terminalLength.encode('utf-8') + b"\n")
                telConnection.write(commands_ShowConfig.encode('utf-8') + b"\n")
            elif(mode[-1] == ":"):
                outputData = b"Password Wrong"
            elif(mode[-1] == ">"):
                outputData = b"Not Enable Mode"


            #sending terminal length command to get full configuration
            #telConnection.write(commands_terminalLength.encode('utf-8') + b"\n")
            #telConnection.write(commands_ShowConfig.encode('utf-8') + b"\n")

            #read till end, so passing a string which may not appear in the configuration
            outputData = telConnection.read_until('stringNotInConfiguration'.encode('utf-8'), timeout=5)

            #close telnet connection
            telConnection.close()
        except:
            outputData = b"telnet not available"
    else:
        outputData = b"Host not pinging"

    #exporting the data to text file and close file
    Write_Info_toFile(directoryLocation, hostIP, outputData)

    return (indexCount, hostIP)
#end Initiate_Telnet


###This method writes info to file
def Write_Info_toFile(directory, fileName, dataString):
    datestring = datetime.datetime.now()
    datestring = str(datestring).replace(':','_').replace('-','_').replace(' ','__').split('.')[0]
    
    #exporting the data to text file and close file
    dataString = dataString.decode('utf-8')
    
    #validation
    if(dataString == "telnet not available"):
        text_file = open(directory + "Log_backup_.txt", "a+")
        dataString = fileName + " " + dataString + "\n"
    elif(dataString == "Host not pinging"):
        text_file = open(directory + "Log_backup_.txt", "a+")
        dataString = fileName + " " + dataString + "\n"
    elif(dataString == ""):
        text_file = open(directory + "Log_backup_.txt", "a+")
        dataString = fileName + " Login Failed" + "\n"
    else:
        text_file = open(directory + fileName + "_backup_" + datestring + ".txt", "w")

    text_file.write(dataString)
    text_file.close()
#end Write_Info_toFile

#start function
def Start_Telnet(deviceFilePath = "D:/Script/DeviceIp.csv", directoryLocation = "D:/Script/"):
    with open(deviceFilePath) as csvDeviceDetails:
        csdData = csv.reader(csvDeviceDetails, delimiter=',')
        #print(csdData)
        lineCount = 0
        for (ipAddress, userName, password) in csdData:
            if(lineCount == 0):
                #First entry is heading
                lineCount += 1
            else:
                lineCount += 1
                Initiate_Telnet(0, ipAddress, userName, password, directoryLocation)

Start_Telnet("D:/Script/DeviceIp.csv", "D:/Script/")

Note: This works for switches which has login to “enable mode”, for other modes, extra conditions has to be added.

Hope you liked this article and thank you for reading