cmc/cleberg.net

  1#+date:        [2020-07-26 Sun 00:00:00]
  2#+title:       Data Analysis: Finding the Best Location for a Business
  3#+description: Part of my data analysis course, I built an algorithm to find the best location for a new business in Lincoln.
  4#+slug:        business-analysis
  5#+filetags:    :audit:
  6
  7* Background Information
  8
  9This project aims to help investors learn more about a random city in order to
 10determine optimal locations for business investments. The data used in this
 11project was obtained using Foursquare's developer API.
 12
 13Fields include:
 14
 15- Venue Name
 16- Venue Category
 17- Venue Latitude
 18- Venue Longitude
 19
 20There are 232 records found using the center of Lincoln as the area of interest
 21with a radius of 10,000.
 22
 23* Import the Data
 24
 25The first step is the simplest: import the applicable libraries. We will be
 26using the libraries below for this project.
 27
 28#+begin_src python
 29# Import the Python libraries we will be using
 30import pandas as pd
 31import requests
 32import folium
 33import math
 34import json
 35from pandas.io.json import json_normalize
 36from sklearn.cluster import KMeans
 37#+end_src
 38
 39To begin our analysis, we need to import the data for this project. The data we
 40are using in this project comes directly from the Foursquare application
 41programming interface (API). The first step is to get the latitude and longitude
 42of the city being studied (Lincoln, NE) and setting up the folium map.
 43
 44#+begin_src python
 45# Define the latitude and longitude, then map the results
 46latitude = 40.806862
 47longitude = -96.681679
 48map_LNK = folium.Map(location=[latitude, longitude], zoom_start=12)
 49
 50map_LNK
 51#+end_src
 52
 53Now that we have defined our city and created the map, we need to go get the
 54business data. The Foursquare API will limit the results to 100 per API call, so
 55we use our first API call below to determine the total results that Foursquare
 56has found. Since the total results are 232, we perform the API fetching process
 57three times (100 + 100 + 32 = 232).
 58
 59#+begin_src python
 60# Foursquare API credentials
 61CLIENT_ID = 'your-client-id'
 62CLIENT_SECRET = 'your-client-secret'
 63VERSION = '20180604'
 64
 65# Set up the URL to fetch the first 100 results
 66LIMIT = 100
 67radius = 10000
 68url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
 69    CLIENT_ID,
 70    CLIENT_SECRET,
 71    VERSION,
 72    latitude,
 73    longitude,
 74    radius,
 75    LIMIT)
 76
 77# Fetch the first 100 results
 78results = requests.get(url).json()
 79
 80# Determine the total number of results needed to fetch
 81totalResults = results['response']['totalResults']
 82totalResults
 83
 84# Set up the URL to fetch the second 100 results (101-200)
 85LIMIT = 100
 86offset = 100
 87radius = 10000
 88url2 = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}&offset={}'.format(
 89    CLIENT_ID,
 90    CLIENT_SECRET,
 91    VERSION,
 92    latitude,
 93    longitude,
 94    radius,
 95    LIMIT,
 96    offset)
 97
 98# Fetch the second 100 results (101-200)
 99results2 = requests.get(url2).json()
100
101# Set up the URL to fetch the final results (201 - 232)
102LIMIT = 100
103offset = 200
104radius = 10000
105url3 = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}&offset={}'.format(
106    CLIENT_ID,
107    CLIENT_SECRET,
108    VERSION,
109    latitude,
110    longitude,
111    radius,
112    LIMIT,
113    offset)
114
115# Fetch the final results (201 - 232)
116results3 = requests.get(url3).json()
117#+end_src
118
119* Clean the Data
120
121Now that we have our data in three separate dataframes, we need to combine them
122into a single dataframe and make sure to reset the index so that we have a
123unique ID for each business. The =get_category_type= function below will pull
124the categories and name from each business's entry in the Foursquare data
125automatically. Once all the data has been labeled and combined, the results are
126stored in the =nearby_venues= dataframe.
127
128#+begin_src python
129# This function will extract the category of the venue from the API dictionary
130def get_category_type(row):
131    try:
132        categories_list = row['categories']
133    except:
134        categories_list = row['venue.categories']
135
136    if len(categories_list) == 0:
137        return None
138    else:
139        return categories_list[0]['name']
140
141# Get the first 100 venues
142venues = results['response']['groups'][0]['items']
143nearby_venues = json_normalize(venues)
144
145# filter columns
146filtered_columns = ['venue.name', 'venue.categories', 'venue.location.lat', 'venue.location.lng']
147nearby_venues = nearby_venues.loc[:, filtered_columns]
148
149# filter the category for each row
150nearby_venues['venue.categories'] = nearby_venues.apply(get_category_type, axis=1)
151
152# clean columns
153nearby_venues.columns = [col.split(".")[-1] for col in nearby_venues.columns]
154
155---
156
157# Get the second 100 venues
158venues2 = results2['response']['groups'][0]['items']
159nearby_venues2 = json_normalize(venues2) # flatten JSON
160
161# filter columns
162filtered_columns2 = ['venue.name', 'venue.categories', 'venue.location.lat', 'venue.location.lng']
163nearby_venues2 = nearby_venues2.loc[:, filtered_columns]
164
165# filter the category for each row
166nearby_venues2['venue.categories'] = nearby_venues2.apply(get_category_type, axis=1)
167
168# clean columns
169nearby_venues2.columns = [col.split(".")[-1] for col in nearby_venues.columns]
170nearby_venues = nearby_venues.append(nearby_venues2)
171
172---
173
174# Get the rest of the venues
175venues3 = results3['response']['groups'][0]['items']
176nearby_venues3 = json_normalize(venues3) # flatten JSON
177
178# filter columns
179filtered_columns3 = ['venue.name', 'venue.categories', 'venue.location.lat', 'venue.location.lng']
180nearby_venues3 = nearby_venues3.loc[:, filtered_columns]
181
182# filter the category for each row
183nearby_venues3['venue.categories'] = nearby_venues3.apply(get_category_type, axis=1)
184
185# clean columns
186nearby_venues3.columns = [col.split(".")[-1] for col in nearby_venues3.columns]
187
188nearby_venues = nearby_venues.append(nearby_venues3)
189nearby_venues = nearby_venues.reset_index(drop=True)
190nearby_venues
191#+end_src
192
193* Visualize the Data
194
195We now have a complete, clean data set. The next step is to visualize this data
196onto the map we created earlier. We will be using folium's =CircleMarker()=
197function to do this.
198
199#+begin_src python
200# add markers to map
201for lat, lng, name, categories in zip(nearby_venues['lat'], nearby_venues['lng'], nearby_venues['name'], nearby_venues['categories']):
202    label = '{} ({})'.format(name, categories)
203    label = folium.Popup(label, parse_html=True)
204    folium.CircleMarker(
205        [lat, lng],
206        radius=5,
207        popup=label,
208        color='blue',
209        fill=True,
210        fill_color='#3186cc',
211        fill_opacity=0.7,
212        ).add_to(map_LNK)
213
214map_LNK
215#+end_src
216
217* Clustering: /k-means/
218
219To cluster the data, we will be using the /k-means/ algorithm. This algorithm is
220iterative and will automatically make sure that data points in each cluster are
221as close as possible to each other, while being as far as possible away from
222other clusters.
223
224However, we first have to figure out how many clusters to use (defined as the
225variable 'k'). To do so, we will use the next two functions to calculate the sum
226of squares within clusters and then return the optimal number of clusters.
227
228#+begin_src python
229# This function will return the sum of squares found in the data
230def calculate_wcss(data):
231    wcss = []
232    for n in range(2, 21):
233        kmeans = KMeans(n_clusters=n)
234        kmeans.fit(X=data)
235        wcss.append(kmeans.inertia_)
236
237    return wcss
238
239# Drop 'str' cols so we can use k-means clustering
240cluster_df = nearby_venues.drop(columns=['name', 'categories'])
241
242# calculating the within clusters sum-of-squares for 19 cluster amounts
243sum_of_squares = calculate_wcss(cluster_df)
244
245# This function will return the optimal number of clusters
246def optimal_number_of_clusters(wcss):
247    x1, y1 = 2, wcss[0]
248    x2, y2 = 20, wcss[len(wcss)-1]
249
250    distances = []
251    for i in range(len(wcss)):
252        x0 = i+2
253        y0 = wcss[i]
254        numerator = abs((y2-y1)*x0 - (x2-x1)*y0 + x2*y1 - y2*x1)
255        denominator = math.sqrt((y2 - y1)**2 + (x2 - x1)**2)
256        distances.append(numerator/denominator)
257
258    return distances.index(max(distances)) + 2
259
260# calculating the optimal number of clusters
261n = optimal_number_of_clusters(sum_of_squares)
262#+end_src
263
264Now that we have found that our optimal number of clusters is six, we need to
265perform k-means clustering. When this clustering occurs, each business is
266assigned a cluster number from 0 to 5 in the dataframe.
267
268#+begin_src python
269# set number of clusters equal to the optimal number
270kclusters = n
271
272# run k-means clustering
273kmeans = KMeans(n_clusters=kclusters, random_state=0).fit(cluster_df)
274
275# add clustering labels to dataframe
276nearby_venues.insert(0, 'Cluster Labels', kmeans.labels_)
277#+end_src
278
279Success! We now have a dataframe with clean business data, along with a
280cluster number for each business. Now let's map the data using six
281different colors.
282
283#+begin_src python
284# create map with clusters
285map_clusters = folium.Map(location=[latitude, longitude], zoom_start=12)
286colors = ['#0F9D58', '#DB4437', '#4285F4', '#800080', '#ce12c0', '#171717']
287
288# add markers to the map
289for lat, lng, name, categories, cluster in zip(nearby_venues['lat'], nearby_venues['lng'], nearby_venues['name'], nearby_venues['categories'], nearby_venues['Cluster Labels']):
290    label = '[{}] {} ({})'.format(cluster, name, categories)
291    label = folium.Popup(label, parse_html=True)
292    folium.CircleMarker(
293        [lat, lng],
294        radius=5,
295        popup=label,
296        color=colors[int(cluster)],
297        fill=True,
298        fill_color=colors[int(cluster)],
299        fill_opacity=0.7).add_to(map_clusters)
300
301map_clusters
302#+end_src
303
304* Investigate Clusters
305
306Now that we have figured out our clusters, let's do a little more analysis to
307provide more insight into the clusters. With the information below, we can see
308which clusters are more popular for businesses and which are less popular. The
309results below show us that clusters 0 through 3 are popular, while clusters 4
310and 5 are not very popular at all.
311
312#+begin_src python
313# Show how many venues are in each cluster
314color_names = ['Dark Green', 'Red', 'Blue', 'Purple', 'Pink', 'Black']
315for x in range(0,6):
316    print("Color of Cluster", x, ":", color_names[x])
317    print("Venues found in Cluster", x, ":", nearby_venues.loc[nearby_venues['Cluster Labels'] == x, nearby_venues.columns[:]].shape[0])
318    print("---")
319#+end_src
320
321Our last piece of analysis is to summarize the categories of businesses within
322each cluster. With these results, we can clearly see that restaurants, coffee
323shops, and grocery stores are the most popular.
324
325#+begin_src python
326# Calculate how many venues there are in each category
327# Sort from largest to smallest
328temp_df = nearby_venues.drop(columns=['name', 'lat', 'lng'])
329
330cluster0_grouped = temp_df.loc[temp_df['Cluster Labels'] == 0].groupby(['categories']).count().sort_values(by='Cluster Labels', ascending=False)
331cluster1_grouped = temp_df.loc[temp_df['Cluster Labels'] == 1].groupby(['categories']).count().sort_values(by='Cluster Labels', ascending=False)
332cluster2_grouped = temp_df.loc[temp_df['Cluster Labels'] == 2].groupby(['categories']).count().sort_values(by='Cluster Labels', ascending=False)
333cluster3_grouped = temp_df.loc[temp_df['Cluster Labels'] == 3].groupby(['categories']).count().sort_values(by='Cluster Labels', ascending=False)
334cluster4_grouped = temp_df.loc[temp_df['Cluster Labels'] == 4].groupby(['categories']).count().sort_values(by='Cluster Labels', ascending=False)
335cluster5_grouped = temp_df.loc[temp_df['Cluster Labels'] == 5].groupby(['categories']).count().sort_values(by='Cluster Labels', ascending=False)
336
337# show how many venues there are in each cluster (> 1)
338with pd.option_context('display.max_rows', None, 'display.max_columns', None):
339    print("\n\n", "Cluster 0:", "\n", cluster0_grouped.loc[cluster0_grouped['Cluster Labels'] > 1])
340    print("\n\n", "Cluster 1:", "\n", cluster1_grouped.loc[cluster1_grouped['Cluster Labels'] > 1])
341    print("\n\n", "Cluster 2:", "\n", cluster2_grouped.loc[cluster2_grouped['Cluster Labels'] > 1])
342    print("\n\n", "Cluster 3:", "\n", cluster3_grouped.loc[cluster3_grouped['Cluster Labels'] > 1])
343    print("\n\n", "Cluster 4:", "\n", cluster4_grouped.loc[cluster4_grouped['Cluster Labels'] > 1])
344    print("\n\n", "Cluster 5:", "\n", cluster5_grouped.loc[cluster5_grouped['Cluster Labels'] > 1])
345#+end_src
346
347* Discussion
348
349In this project, we gathered location data for Lincoln, Nebraska, USA and
350clustered the data using the k-means algorithm in order to identify the unique
351clusters of businesses in Lincoln. Through these actions, we found that there
352are six unique business clusters in Lincoln and that two of the clusters are
353likely unsuitable for investors. The remaining four clusters have a variety of
354businesses, but are largely dominated by restaurants and grocery stores.
355
356Using this project, investors can now make more informed decisions when deciding
357the location and category of business in which to invest.
358
359Further studies may involve other attributes for business locations, such as
360population density, average wealth across the city, or crime rates. In addition,
361further studies may include additional location data and businesses by utilizing
362multiple sources, such as Google Maps and OpenStreetMap.