How To Load And Save Data In Python

Every bit a beginner, you might only know a single way to load data (normally in CSV) which is to read it using p andas.read_csv function. It is ane of the most mature and strong functions, but other means are a lot helpful and volition definitely come up in handy sometimes.

The ways that I am going to talk over are:

Manualfunction
loadtxtpart
genfromtxt function
read_csvfunction
Pickle

The dataset that we are going to utilise to load data can be institutehither. It is named as 100-Sales-Records.

Imports

Nosotros volition use Numpy, Pandas, and Pickle packages so import them.

import numpy equally np import pandas as pd import pickle

1. Manual Function

This is the about hard, as you have to design a custom function, which can load data for you. Yous have to deal with Python's normal filing concepts and using that you lot have to read a.csv file.

Let's do that on 100 Sales Records file.

def load_csv(filepath):     data =  []     col = []     checkcol = Faux     with open(filepath) as f:         for val in f.readlines():             val = val.supersede("\north","")             val = val.split(',')             if checkcol is False:                 col = val                 checkcol = True             else:                 data.append(val)     df = pd.DataFrame(information=data, columns=col)     return df

Hmmm, What is this????? Seems a bit complex lawmaking!!!! Let'south intermission information technology stride by step and so you know what is happening and you tin employ similar logic to read a.csv file of your ain.

Here, I have created aload_csv a part that takes in as an statement the path of the file you want to read.

I accept a list named asdatawhich is going to have my data of CSV file, and another listingcolwhich is going to take my cavalcade names. Now afterwards inspecting the csv manually, I know that my column names are in the first row, and then in my first iteration, I have to shop the data of the get-go row incoland rest rows indata.

To check the offset iteration, I take used a Boolean Variable named ascheckcolwhich is False, and when it is false in the first iteration, it stores the data of first-line incoland so it setscheckcolto True, so we will deal withinformation list and store rest of values ininformation listing.

Logic

The main logic here is that I have iterated in the file, usingreadlines() a function in Python. This function returns a list that contains all the lines inside a file.

When reading through headlines, information technology detects a new line as\n grapheme, which is line terminating character, so in order to remove it, I have usedstr.replace function.

As it is a.csv file, so I accept to carve up things based oncommasand then I will carve up the string on a , usingcord.split(','). For the commencement iteration, I will store the starting time row, which contains the column names in a listing known equallycol. And then I will append all my data in my list known every bitdata.

To read the data more beautifully, I take returned it every bit a dataframe format considering it is easier to read a dataframe as compared to a numpy array or python's list.

Output

myData = load_csv('100 Sales Tape.csv') impress(myData.head())

Information from Custom Function.

Pros and Cons

The important do good is that yous accept all the flexibility and control over the file structure and y'all can read in whatever format and mode you want and store it.

You can also read the files which do not have a standard structure using your own logic.

Of import drawbacks of it are that it is complex to write especially for standard types of files because they can hands be read. You take to hard lawmaking the logic which requires trial and fault.

You should merely utilise it when the file is not in a standard format or you want flexibility and reading the file in a way that is non available through libraries.

2. Numpy.loadtxt function

This is a congenital-in office in Numpy, a famous numerical library in Python. It is a really simple office to load the data. It is very useful for reading data which is of the same datatype.

When data is more circuitous, it is hard to read using this role, just when files are easy and uncomplicated, this role is really powerful.

To get the data of a single blazon, you lot can download this dummy dataset. Let's jump to code.

df = np.loadtxt('convertcsv.csv', delimeter = ',')

Here we simply used theloadtxtfunction as passed indelimeteras',' because this is a CSV file.

Now if we printdf, we will see our data in pretty decent numpy arrays that are fix to utilise.

We take just printed the starting time five rows due to the big size of data.

Pros and Cons

An important aspect of using this function is that you can quickly load in information from a file into numpy arrays.

Drawbacks of information technology are that y'all can not have different information types or missing rows in your data.

3. Numpy.genfromtxt()

Nosotros will utilise the dataset, which is '100 Sales Records.csv' which we used in our first example to demonstrate that we can have multiple data types in it.

Let's jump to code.

information = np.genfromtxt('100 Sales Records.csv', delimiter=',')

and to see it more clearly, we can just come across it in a dataframe format, i.e.,

Wait? What is this? Oh, Information technology has skipped all the columns with string information types. How to bargain with it?

Just add some otherdtypeparameter and setdtypeto None which means that it has to accept intendance of datatypes of each column itself. Not to convert whole data to single dtype.

data = np.genfromtxt('100 Sales Records.csv', delimiter=',', dtype=None)

And and so for output:

>>> pd.DataFrame(data).head()

Quite ameliorate than the start one, but here our Columns titles are Rows, to make them cavalcade titles, we have to add another parameter which isnamesand gear up it toTruthfulso it volition take the outset row as the Cavalcade Titles.

i.due east.

information = np.genfromtxt('100 Sales Records.csv', delimiter=',', dtype=None, names = Truthful)

and we tin can impress it as:

>>> pd.DataFrame(df3).head()

And here we tin run into that It has successfully added the names of columns in the dataframe.

At present the last problem is that the columns which are of cord data types are non the bodily strings, simply they are inbytesformat. You can see that before every string, we accept ab' so to encounter them, nosotros have to decode them in utf-8 format.

df3 = np.genfromtxt('100 Sales Records.csv', delimiter=',', dtype=None, names=Truthful, encoding='utf-8')

This will return our dataframe in the desired form.

4. Pandas.read_csv()

Pandas is a very pop data manipulation library, and it is very usually used. 1 of it's very important andmaturefunctions isread_csv() which can read whatsoever.csvfile very easily and assistance us manipulate it. Permit's do information technology on our 100-Sales-Record dataset.

This function is very popular due to its ease of use. You tin can compare it with our previous codes, and you lot can bank check it.

>>> pdDf = pd.read_csv('100 Sales Record.csv') >>> pdDf.caput()

And judge what? We are done. This was actually then simple and like shooting fish in a barrel to employ. Pandas.read_csv definitely offers a lot of other parameters to tune in our data set, for example in ourconvertcsv.csv file, we had no column names and then we can read it equally:

>>> newdf = pd.read_csv('convertcsv.csv', header=None) >>> newdf.head()

And we can see that it has read thecsvfile without the header. You can explore all other parameters in the official docs hither.

five. Pickle

When your information is not in a good, human being-readable format, you can utilise pickle to save it in a binary format. Then you can easily reload it using the pickle library.

We will have our 100-Sales-Record CSV file and kickoff relieve information technology in a pickle format and so nosotros can read it.

with open('test.pkl','wb') as f:     pickle.dump(pdDf, f)

This will create a new fileexamination.pkl which has inside it ourpdDffromPandas heading.

Now to open it using pickle, nosotros just have to usepickle.load role.

with open("test.pkl", "rb") as f:     d4 = pickle.load(f)  >>> d4.head()

And hither we have successfully loaded data from a pickle file inpandas.DataFrameformat.

You are at present aware of v different ways to load data files in Python, which tin can help you in unlike ways to load a data set up when you are working in your solar day-to-solar day projects.