Web scraping and using twitter API to analyze user data

This is my very first blog post and am happy for it because it’s all about my starting journey in data science. My gratitude is to 10 Academy For they have indeed shaping me into a job ready data scientist through every week challenge. This blog will be about my first week challenge report.

To start with the challenge I had first to get some html data from two websites, that is called Web scraping. To do this there are two amazing packages that I used.

The websites from which I scraped data are:

The websites has names of users which provided a starting point for getting their twitter information ,analyze them and compare their influence measure.

After installing and importing the libraries needed ,the next step was to get the data from the websites.

In the above codes scraps the data from the first website, the beautiful soups package convert every page content which is an <h2> tag into a list of strings. for our website the top influencers are in tag h2.

For accessing the twitter API, a twitter developer account is required to have authorization. Moreover a tweepy ( — A python wrapper for interacting with twitter API)need to be installed.

!pip install tweepy

Next I created an authorization object using below codes:

For making the analysis easy and for influence measuring I was interested in getting the:

Now after getting that I had to find the number of times a user was mentioned in some tweets which I did in this way:

Now all needed information information of users are obtained, I was on my way to do the analysis.

From the activities of a user in twitter, we get three types of influence as outlined in earlier mention document; Million follower fallacy, These are:

As it was hard for me to determine the comments I have decided to use only mentions for relevance score.

After I sorted the reach score and got the top10 influencers, then I plotted the result.

After analyzing the data the following was drawn out:

Retweet are driven by the content value of a tweet, while mentions are are driven by name value of a user. Hence a recommendation for ordinary users to post rich and creative content to gain influence.

For companies which wants to start a digital campaign I can recommend them to work with the covid19 related hashtags also can make the campaign achievable

The following materials were very useful during the exercise:

Related posts:

Hi my name is Dave Chappell and in 2020 our family is about to embark on a new adventure. The business will be called Chappell Cosmetics and this my journey into a female dominated industry. In 2014…

Learn how to treat acne naturally! These 5 natural acne treatments are ways to keep your skin clear, blemish-free & make your skin look bright and glowing! Let’s face it… most of us deal with acne at…

Just as there are many kinds of garage designs, this garage guide classifies, explores and breaks many Garage option you’ve got for your residence. I have lived without a garage and now possess a…