When you join two DFs with similar column names:
df = df1.join(df2, df1['id'] == df2['id'])
Join works fine but you can't call the
id column because it is ambiguous and you would get the following exception:
pyspark.sql.utils.AnalysisException: "Reference 'id' is ambiguous, could be: id#5691, id#5918.;"
id not usable anymore...
The following function solves the problem:
def join(df1, df2, cond, how='left'): df = df1.join(df2, cond, how=how) repeated_columns = [c for c in df1.columns if c in df2.columns] for col in repeated_columns: df = df.drop(df2[col]) return df
What I don't like about it is that I have to iterate over the column names and delete them why by one. This looks really clunky...
Do you know of any other solution that will either join and remove duplicates more elegantly or delete multiple columns without iterating over each of them?