I'm working with a DataFrame in Python using pandas, and I'm trying to apply multiple conditions to filter rows based on temperature values from multiple columns. However, after applying my conditions and using dropna()
, I end up with zero rows even though I expect some data to meet these conditions.
The goal is compare with Ambient temp+40 C and if the value is more than this, replace it with NaN. Otherwise, keep the original value.
Here's a sample of my DataFrame and the conditions I'm applying:
data = {
'Datetime': ['2022-08-04 15:06:00', '2022-08-04 15:07:00', '2022-08-04 15:08:00',
'2022-08-04 15:09:00', '2022-08-04 15:10:00'],
'Temp1': [53.4, 54.3, 53.7, 54.3, 55.4],
'Temp2': [57.8, 57.0, 87.0, 57.2, 57.5],
'Temp3': [59.0, 58.8, 58.7, 59.1, 59.7],
'Temp4': [46.7, 47.1, 80, 46.9, 47.3],
'Temp5': [52.8, 53.1, 53.0, 53.1, 53.4],
'Temp6': [50.1, 69, 50.3, 50.3, 50.6],
'AmbientTemp': [29.0, 28.8, 28.6, 28.7, 28.9]
}
df1 = pd.DataFrame(data)
df1['Datetime'] = pd.to_datetime(df1['Datetime'])
df1.set_index('Datetime', inplace=True)
Code:
temp_cols = ['Temp1', 'Temp2', 'Temp3', 'Temp4', 'Temp5', 'Temp6']
ambient_col = 'AmbientTemp'
condition = (df1[temp_cols].lt(df1[ambient_col] + 40, axis=0))
filtered_df = df1[condition].dropna()
print(filtered_df.shape)
Response:
(0, 99)
Problem:
Despite expecting valid data that meets the conditions, the resulting DataFrame is empty after applying the filter and dropping NaN values. What could be causing this issue, and how can I correct it?