pandas performance optimization

vectorization:
  wrong: df['new_col'] = [x * 2 for x in df['col']]
  correct: df['new_col'] = df['col'] * 2

avoid loops:
  wrong: for i in range(len(df)): df.loc[i, 'new'] = df.loc[i, 'old'] * 2
  correct: df['new'] = df['old'] * 2

use built-in methods:
  wrong: df['col'].apply(lambda x: x.strip().lower())
  correct: df['col'].str.strip().str.lower()

optimize dtypes:
  df['id'] = df['id'].astype('int32')  # instead of int64
  df['category'] = df['category'].astype('category')

chunk processing for large files:
  chunk_size = 10000
  for chunk in pd.read_csv('large.csv', chunksize=chunk_size):
      process(chunk)

use inplace carefully:
  df.drop(columns=['unused'], inplace=True)  # saves memory
  df.sort_values('date', inplace=True)  # avoids copy
