I have never used .pipe() before but I had similar experience of when I finally discovered the .shift(). (I guess I should read the documentation more).

I think the one thing I am confused about is why are the functions needed here for single lines of code that are hardcoded?

Wouldn't this work the same:

df_final = df[df["price"] > 100]
df_final["total_value"] = df_final["price"] * df_final["quantity"]
df_final = df_final.sort_values(by="total_value", ascending=False]

# or keeping the old dataframe in tact
df_final = df[df["price"] > 100].copy()
df_final["total_value"] = df_final["price"] * df_final["quantity"]
df_final = df_final.sort_values(by="total_value", ascending=False]

You can still see line by line what is happening and you don't have to trace back to functions located somewhere else. I know this is an example, but are the ETL functions more complex that are stored in a separate file/module?

I know aesthetics are subjective, so I won't argue that debate on what people prefer. I can say I prefer just writing the three lines instead of searching through functions to figure out what they do (unless a function is truly required). If you have issues you can still comment out a single line of code and debug.

To me, .pipe() might be more useful if it were passing in arguments that could change, or if you were modifying multiple parts of the dataframe. I am thinking if all these steps were in one function, or if parameters needed to be passed into the function

def my_pipeline(df):
  df = df[df["price"] > 100]
  df["total_value"] = df["price"] * df["quantity"]
  df = df.sort_values(by="total_value", ascending=False]
  return df

def filter_by_price(df, price):
  return df[df["price"] > price]

# or even more genearlized
# filter by greater then value for any column
df filter_column_gt_value(df, column_name, price):
  return df[df[column_name] > price]

# and you could do the same for the other comparison operaators
df filter_column_lt_value(df, column_name, price):
  return df[df[column_name] < price]

# now these functions are more generalized and portable
# but can also be more difficult to parse through and read
# it's a spaghetti!

Otherwise, I like to use apply if I am applying a function to a single column. For example, I don't like the way pandas calculates years between today's date and a previous date, so I created my own and use .apply(), mainly for calculating ages.

def calculate_years_between(end_date: pd.Timesamp) -> int:
  today = pd.Timestamp.today()
  if today.month < end_date.month:
    return today.year - end_date.year - 1
  if today.month == end_date.month & today.day < end_date.day:
    return today.year - end_date.year - 1
  return today.year - end_date.year

df['age'] = df['birthday'].apply(calculate_years_between)

I don't know if there are performance benefits to using .pipe() over other ways (memory or speed wise)? So, if anyone can shed some light on that, it would be great.