Pandas

Filter

1. filter with multiple condition

   df_filtered = df[ (df.D>0) & df.C<0) ]
   df_filtered = df[ (df['D']>0) & df['C']<0) ]
   

2. filter with isin

   alist = [12,312,53]
   df_filtered = df[ df['D'].isin(alist) ]
   

3. Ref

   • http://jingyan.baidu.com/article/0eb457e508b6d303f0a90572.html
   • pandas.DataFrame.filter

Indicator of apply when using pandas

1. using the tqdm model

   originally:

   df.groupby(['feature1', 'feature2']).apply(feature_rollup)
   

   using tqdm:

   from tqdm import tqdm
   tqdm.pandas(tqdm())
   df.groupby(['feature1', 'feature2']).progress_apply(feature_rollup)
   

2. Ref

   • https://stackoverflow.com/questions/18603270/progress-indicator-during-pandas-operations-python

Handle Large scale data

1. Ref
   • https://read01.com/zh-tw/4084.html#.WakUlpOg9hE

Melting data frames

1. Ref
   • https://gist.github.com/lzhbrian/c2a7473ce5d11ed5704ee1764f4d53aa