Json Python
Json is a very popular data format and is widely used for data exchange. JSON is human readable format. It is very easy to parse Json data in Python. In Python, we can read the Json data either from a file or from an API. In this post, I will go over some of the most common commands to read, parse, structure and write Json data.
Python has Json package. Let us import that.
import json
Let us build a sample Json data. In the below example, I have constructed a Json data. Json is all about keys and values. In our json example below, we have two main keys - 'state' and 'citites'. Values can be one or multiple values. For multiple values, we can mention values in square brackets. In python terms values inside square brackets is called list.
One important thing to notice here is that, in the below example all the keys and values are in double quotes. You can't have single quotes. We can mix and match strings and numbers though.
Python Load Json
x = {"state":"california","cities":["los angeles","san francisco"]}
Ok I have created a json object x. The variable x is a dictionary. To convert in to Json. We need to first convert above dictionary to Json string format using json.dumps method.
type(json.dumps(x))
d = json.dumps(x)
Ok as we see above, we get a str object but of type Json. To convert it back to Python dictionary, we can use json.loads() method.
y = json.loads(json.dumps(x))
Note y is a dictionary in Python so we can access the keys and values just like we do for regular dictionaries in Python as shown below.
y['state']
type(json.loads(json.dumps(x)))
Python read Json from API Url
Ok in Python we can read Json directly from an API. For this we need Python Pandas. If you don't know about Pandas. Read about pandas.
Let us first import Python pandas package.
import pandas as pd
Let us read the options data from following yahoo finance API using pd.read_json() function.
jsondata = pd.read_json('https://query2.finance.yahoo.com/v7/finance/options/amzn')
type(jsondata)
Ok as we see above, by default, pandas creates a DataFrame. Json data can be quite complex and can contain multiple nested key values pairs and therefore can become very long. Dataframe is not the right data structure to analyze the json data. Therefore we need to convert this dataframe to Python dictionary first using to_dict() method as shown below.
data = jsondata.to_dict()
Let us take a look into our data now.
data.keys()
Ok, as we see above, the main key is 'optionChain' which contains all the data. We can access and loop through above dictionary as we do normally in Python dictionaries.
data['optionChain']['result'][0]['expirationDates'][0]
Write Json data to file in Python
To write json data to file, we can use either Json package or Pandas package. Let us first try the Pandas way.
Pandas has to_json() method. Let us apply this to our jsondata Pandas dataframe which has options data for one stock.
with open('optionsdata.json', 'w') as fp:
fp.write(jsondata.to_json())
ls -lrt optionsdata.json
Ok now let us try the 2nd way which is by using the json package
with open('optionsdata1.json', 'w') as fp:
fp.write(json.dumps(jsondata.to_dict()))
ls -lrt optionsdata1.json
Read Json data from a file in Python
Reading Json data from a file in Python is pretty straight forward. Let us read back the json file that we just wrote. Let us use the Pandas read_json function first.
jodata = pd.read_json('optionsdata.json')
type(jodata)
Then as usual, use jodata.to_dict() for further processing in Python.
We can also use json package's json.loads() to read the json file as shown below.
with open('optionsdata.json', 'r') as fp:
jodata = json.loads(fp.read())
type(jodata)
Json Array or List in Python
Json List in python is a list of Json rows. Each row in itself is a valid json doc or row. Let us take above options data for couple of stocks and create a Python Json list.
stocks = ['nvda','intc']
jsonlist = []
for stock in stocks:
jsondata = pd.read_json('https://query2.finance.yahoo.com/v7/finance/options/%s'%stock)
jsonlist.append(jsondata)
len(jsonlist)
Ok, as we see above. We have two entries of Json data in our jsonlist variable. Note, we have used pd.read_json method to read directly from the API url. If you remember, each json row or entry is a Pandas dataframe. So will have to use to_dict() method.
type(jsonlist[0])
json.loads(json.dumps(jsonlist))
If we run the above, we would get following error. Reason, jsonlist is a python list.
TypeError: Object of type DataFrame is not JSON serializable
Ofcourse we can do following...
jodata = jsonlist[0].to_dict()
type(jodata)
To convert all rows to dictionary,we can loop through the jsonlist.
How to write Json Array or List to a file in Python using Pandas
Ok let us take our jsonlist and write to a file. Remember each row or entry in the jsonlist is a Pandas Dataframe. So we need to convert each entry to json first before writing to file.
Let us try to write Ist json document.
with open('stocks_data.json','w') as fp:
fp.write(jsonlist[0].to_json())
ls -lrt stocks_data.json
Let us write now all the entries of jsonlist to a file now.
with open('stocks_data.json','w') as fp:
for entry in jsonlist[0:2]:
fp.write(entry.to_json())
fp.write('\n')
Note, we have added a new line '\n' that way each line will have one json entry.
Let us try to read back the Json list or Array from the file, we just created above.
jsonlist = []
with open('stocks_data.json','r') as fp:
for line in fp:
jsonlist.append(json.loads(line))
len(jsonlist)
jsonlist[0]['optionChain']['result'][0]['expirationDates'][0]
How to write Json Array or List to a file in Python using json package
Let us now try writing json Array to a file using json package.
We need to check whether the entries in jsonlist are Python dictionary or a json string.
type(jsonlist[0])
Ok, now we have the each entry as dictionary, so we will have to use the json.dump method as shown below.
with open('stocks_data.json','w') as fp:
for entry in jsonlist:
json.dump(entry,fp)
ls -lrt stocks_data.json
Wrap Up!
I have tried to cover the basics of Json handling in Python with examples. I hope, this will help you understanding Json parsing in Python.
Related Notebooks
- How To Read JSON Data Using Python Pandas
- How to do SQL Select and Where Using Python Pandas
- Summarising Aggregating and Grouping data in Python Pandas
- Merge and Join DataFrames with Pandas in Python
- How to Convert Python Pandas DataFrame into a List
- How to Plot a Histogram in Python
- How to Generate Random Numbers in Python
- Python Pandas String To Integer And Integer To String DataFrame
- How to Upgrade Python PIP