Cleaning Mammalian and Avian Species Names

Data collected on species can be erroneous and not standardize. Standardization of species names in the analysis is essential for various methods especially during table joins and merges.

To standardize species names in the data, we can match species names in our data with standard dataset such as the one provided by IUCN.
http://www.iucnredlist.org/technical-documents/spatial-data

Here the vignette shows an example code where species list is matched with IUCN species names using fuzzy matches, which are incomplete or inexact matches.
The Python package fuzzywuzzy has a few functions that can help in this matching. To install the package use
#pip install fuzzywuzzy

Importing packages

pandas for dataframe management
fuzzywuzzy for fuzz matching

import pandas as pd
from fuzzywuzzy import fuzz
from fuzzywuzzy import process

"""Setting data paths"""
data_path = 'C:\Users\Falco\Desktop\directory\Link_Prediction\data'

reading data into pandas dataframe

species_list: dataframe of consisting species names. This is the data which we expect to standardize.
IUCN: reference database with standardized names of species

species_list = pd.read_pickle(data_path+'\Species_list.pkl')
IUCN = pd.read_csv(data_path+ '\IUCN Mammals, Birds, Reptiles, and Amphibians.csv',)
IUCN["ScientificName"] = IUCN["Genus"].map(str) +' '+IUCN["Species"]

species_list.head()

List of Correct Names

The fuzzywuzzy package finds the best matching string from the list and return the string matching ratio along with it

list_of_correct_names = IUCN['ScientificName'].tolist()

Following are wrapping functions around the fuzzywuzzy package to extract best matching name and matching ratio for a pandas dataframe

It returns matching name only of the matching percent is greater than 90%

def fillVName(c):
    name_to_check = c.ScientificName
    a = process.extract(name_to_check, list_of_correct_names, limit=1) 
    if a[0][1]>= 90:
        return a[0][0]
    else:
        return np.nan

def fillVmatch(c):
    name_to_check = c.ScientificName
    a = process.extract(name_to_check,list_of_correct_names, limit=1) 
    if a[0][1]>= 90:
        return a[0][1]
    else:
        return np.nan

%%time
species_list['Matched_ScientificName'] = species_list.apply(fillVName, axis=1)

WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '']

%%time
species_list['ratio'] = species_list.apply(fillVmatch, axis=1)

WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '']


Wall time: 12min 25s

Matching names with matching ratio greater than 95% are generally spelling mistakes.
for example, row number 6, where Spelling in the database was ‘Accipter nisus’ but the correct name is Accipiter nisus (97%) matching.

All the unidentified species with sp in their binomial name gets a NaN value assigned as matching string has ratio less than 90%.

*Some Species have completely changed names either due to taxonomical udates in genus name. For example row 21
*Agouti paca is now called as Cuniculus paca. **

I would manually go and check all the matching entries with less than 95% matching and correct them manually.

species_list.drop('Source', inplace=True, axis=1)
species_list.head(50)

Entries with less than 100% of matching ratio

species_list.ratio.fillna(0, inplace= True)
manual_edit = species_list[species_list.ratio <100]

manual_edit.head()

'there are '+ str(manual_edit.shape[0])+ ' entries which do not have perfect match'

'there are 219 entries which do not have perfect match'

Removing entries with unidentified species names

a = manual_edit[~manual_edit.ScientificName.str.contains(".sp")]
a.head()

'there are '+ str(a.shape[0])+ ' entries which do not have perfect match and are completely identified'

'there are 151 entries which do not have perfect match and are completely identified'

Most likely spelling mistakes
entries with matching ratio higher than 95%

b =a[a.ratio >=95]
b.head(50)

'there are '+ str(b.shape[0])+ ' entries which do not most likely spelling mistakes'

'there are 36 entries which do not most likely spelling mistakes'

Needs detailed investigation
entries with matching ratio lesser than 95%

c =a[a.ratio <95]
c.head()

'there are '+ str(c.shape[0])+ ' entries which need detail investigation'

'there are 115 entries which need detail investigation'

species_list.to_csv(data_path+'Corrected_species_names.csv')