SparkToPandas package — SparkToPandas 1.0.1 documentation

# SparkToPandas package

## Submodules

## SparkToPandas.pandas_plugin module

### SparkToPandas Documentation

SparkToPandas is a simple plugin alongside of spark, the SparkToPandas was designed to work with pyspark with a syntax more similar to pandas.

class SparkToPandas.pandas_plugin.spark_pandas(spark)

  Bases: `object`

  A supporting functions for pyspark ,which has the syntax similar to pandas

  barChart(df, x, y, hue, title, aspect='horizontal')

    Plots a barchart using the seaborn module

    Parameters

      - df – dataframe

      - x – str

      - y – str

      - hue – str

      - title – str

      - aspect – str

    Returns

      None

  column_creator(df, primary_column, new_column_name, user_func)

    Creates a new column based on user defined function and returns the new rdd

    Parameters

      - df – dataframe

      - primary_column – str

      - new_column_name – str

      - user_func – function

    Returns

      dataframe

  conditional_func(x)

    A sample function, to add x+1 number

    Parameters

      x – int

    Returns

      int

  drop_na(df, col_name=None)

    Drops null values based on user choice. Supports dropping all null values or dropping null values based on column subset

    Parameters

      - df – dataframe

      - col_name – str

    Returns

      dataframe

  fillna(df, value, col_name=None)

    Fills null values based on user choice.

    Parameters

      - df – dataframe

      - value – int/str/float

      - col_name – str

    Returns

      dataframe

  head(df, n)

    Prints the head and tail of the dataframe depending on user’s choice.

    Parameters

      - df – dataframe

      - n – int

    Returns

      None

  read_csv(file_location, header=True)

    Function to read csv file as a spark rdd

    Parameters

      - file_location – str

      - header – bool

    Returns

      rdd

  sort_df(df, col_name, ascending=True)

    Function to sort the dataframe in ascending or descending order based on the columns given

    Parameters

      - df – dataframe

      - col_name – list

      - ascending – bool

    Returns

      dataframe

  subset_columns(column_names, df)

    Returns a dataframe which the user specified column names.

    Parameters

      - column_names – list

      - df – dataframe

    Returns

      dataframe

## Module contents