Getting the Data from API

2023/02/19

To test getting the API, the forum topic data we need to use the following. 

f"https://api.myanimelist.net/v2/forum/topic/{topic_id}"

To test out only using one of the IDs that we got from the parser, we used the same libraries: 

import requests 
import json 
import pandas as pd

Example of the result for a simple test was this: 

response = requests.get(f"https://api.myanimelist.net/v2/forum/topic/243721?limit=1", headers=headers)
forum_data = response.json()
print(forum_data)

It's too hard to read....

So put it in the json beautifier! 

https://codebeautify.org/jsonviewer

Since we will be working with multiple IDs in the future, we would need a for loop to get this data 

t_id = ['1123755']

bodies = []
for ids in t_id:
    response = requests.get(f"https://api.myanimelist.net/v2/forum/topic/{ids}?limit=100", headers=headers)
    forum_data = response.json()
    bodies += [item['body'] for item in forum_data['data']['posts']]

Having an empty list `bodies` will keep track of all the data inside the "body" part of the json data, and it will add 100 posts by the users. 

 

The final step is this into a csv file to easily view and modify data using the pandas library.  

# Create a DataFrame from the bodies list
df = pd.DataFrame({'message': bodies})
df.to_csv('forum_data.csv', index=False)

tada!

Now we are done with the 

1. Using the API (setting up the API key)
2. Scrapping the anime IDs for future dataset 
3. Getting 100 posts from a forum post using the API set 

Next we will have to clean the text now, which is for next time~