Q10: Data Wrangling Problem Statement: Data Wrangling on Real Estate Market Dataset: "RealEstate_Prices.csv" Description: The dataset contains information about housing prices in a specific real estate market. It includes various attributes such as property characteristics, location, sale prices, and other relevant features. The goal is to perform data wrangling to gain insights into the factors influencing housing prices and prepare the dataset for further analysis or modeling. Tasks to Perform: 1. Import the "RealEstate_Prices.csv" dataset. Clean column names by removing spaces, special characters, or renaming them for clarity. 2. Handle missing values in the dataset, deciding on an appropriate strategy (e.g., imputation or removal). 3. Perform data merging if additional datasets with relevant information are available (e.g., neighborhood demographics or nearby amenities). 4. Filter and subset the data based on specific criteria, such as a particular time period, property type, or location. 5. Handle categorical variables by encoding them appropriately (e.g., one-hot encoding or label encoding) for further analysis. 6. Aggregate the data to calculate summary statistics or derived metrics such as average sale prices by neighborhood or property type. 7. Identify and handle outliers or extreme values in the data that may affect the analysis or modeling process.

import re import pandas as pd import matplotlib.pyplot as plt homes = pd.read_csv("RealEstate_Prices.csv") amenities = pd.read_csv("RealEstate_Neighborhoods.csv") print("Raw shape:", homes.shape) print(homes.head().to_string(index=False)) print("Missing values:\n", homes.isna().sum().to_string())

# Clean column names and types; fill the missing area with the median. homes.columns = [re.sub(r"[^a-z0-9]+", "_", name.lower()).strip("_") for name in homes.columns] homes["sale_date"] = pd.to_datetime(homes["sale_date"], errors="coerce") homes["neighborhood"] = homes["neighborhood"].astype(str).str.strip().str.title() homes["area_sqft"] = homes["area_sqft"].fillna(homes["area_sqft"].median()) homes = homes.dropna(subset=["sale_date", "sale_price"]) print("Clean columns:", homes.columns.tolist())

# Merge neighborhood amenities and select a recent subset. homes = homes.merge(amenities, left_on="neighborhood", right_on="Neighborhood", how="left", validate="many_to_one") homes = homes.drop(columns="Neighborhood") recent = homes.loc[homes["sale_date"] >= "2024-01-01"].copy() print("Recent properties:", len(recent)) print(recent[["neighborhood", "NearbyAmenities"]].head().to_string(index=False))

# Cap unusually high prices and derive price per square foot. q1, q3 = recent["sale_price"].quantile([0.25, 0.75]) upper = q3 + 1.5 * (q3 - q1) print("Prices above IQR upper bound:", int((recent["sale_price"] > upper).sum())) recent["sale_price_capped"] = recent["sale_price"].clip(upper=upper) recent["price_per_sqft"] = recent["sale_price_capped"] / recent["area_sqft"] print(recent.groupby("neighborhood")["sale_price_capped"].agg(["count", "mean", "median"]).round(2).to_string())

# Encode categorical columns for later modeling and compare neighborhoods. model_features = pd.get_dummies(recent[["neighborhood", "property_type", "area_sqft", "bedrooms", "year_built", "NearbyAmenities"]], drop_first=True, dtype=int) print("Encoded feature columns:", model_features.columns.tolist()) recent.groupby("neighborhood")["sale_price_capped"].mean().sort_values().plot.barh(title="Average sale price by neighborhood") plt.xlabel("Sale price") plt.tight_layout() plt.show()

Data Modeling & Visualization (dmv) Codes

Group Part II: Data Modeling and Visualization

Data Wrangling

Q10: Data Wrangling

Other Questions in Data Modeling & Visualization