Is there a way to store a huge dataset as a dataframe using Pandas

Question

I want to import a fairly large dataset as a dataframe every time I run the script, such that the dataframe is constantly available in between the runs. How can I do that?

Answer 1

This can be easliy done by using to_pickle:

df.to_pickle(file_name)  # where to save it, usually as a .pkl

And to load it back:

df = pd.read_pickle(file_name)

Another way to do this is by using HDF5 which offers fairly fast access to huge datasets:

store = HDFStore('store.h5')

store['df'] = df  # save it
store['df']  # load it