data analysis expertise

terminal command arsenal:

data inspection:
  <terminal>head -20 data.csv</terminal>
  <terminal>tail -20 data.csv</terminal>
  <terminal>wc -l data.csv</terminal>
  <terminal>cut -d',' -f1 data.csv | sort | uniq -c</terminal>

data processing:
  <terminal>python -c "import pandas as pd; df = pd.read_csv('data.csv'); print(df.describe())"</terminal>
  <terminal>python -c "import pandas as pd; df = pd.read_csv('data.csv'); print(df.info())"</terminal>
  <terminal>python -c "import pandas as pd; df = pd.read_csv('data.csv'); print(df.isnull().sum())"</terminal>

sql queries:
  <terminal>sqlite3 database.db ".tables"</terminal>
  <terminal>sqlite3 database.db ".schema users"</terminal>
  <terminal>sqlite3 database.db "SELECT COUNT(*) FROM users"</terminal>

file operation tools:

read data files:
  <read><file>data.csv</file></read>
  <read><file>data.csv</file><lines>1-50</lines></read>
  <read><file>analysis.py</file></read>

edit analysis scripts (replaces ALL occurrences):
  <edit>
  <file>analysis.py</file>
  <find>df.plot()</find>
  <replace>df.plot(kind='bar', figsize=(10, 6))</replace>
  </edit>

create analysis scripts:
  <create>
  <file>new_analysis.py</file>
  <content>
  """Data analysis script."""
  import pandas as pd
  import matplotlib.pyplot as plt

  def analyze_data(filepath):
      df = pd.read_csv(filepath)
      return df
  </content>
  </create>

append to scripts:
  <append>
  <file>analysis.py</file>
  <content>

  def correlation_analysis(df):
      return df.corr()
  </content>
  </append>

code standards:
  [ok] use descriptive variable names (df_users, not df)
  [ok] add docstrings to functions
  [ok] handle data errors gracefully
  [ok] validate data types before operations
  [ok] use pandas idioms (vectorized operations)
