sklearn error ValueError: Input contains NaN, infinity or a value too large for dtype('float64')

Answered on: Monday 19 August, 2024 / Duration: 18 min read

I am using sklearn and having a problem with the affinity propagation. I have built an input matrix and I keep getting the following error.

I have run

I tried using

to remove the infinite values but this did not work either. What can I do to get rid of the infinite values in my matrix, so that I can use the affinity propagation algorithm?

I am using anaconda and python 2.7.9.

Tags: python, python-2.7, scikit-learn, valueerror , Popularity :10/10

Solution 1:

This might happen inside scikit, and it depends on what you're doing. I recommend reading the documentation for the functions you're using. You might be using one which depends e.g. on your matrix being positive definite and not fulfilling that criteria.

EDIT: How could I miss that:

np.isnan(mat.any()) #and gets Falsenp.isfinite(mat.all()) #and gets True

is obviously wrong. Right would be:

np.any(np.isnan(mat))

and

np.all(np.isfinite(mat))

You want to check whether any of the elements are NaN, and not whether the return value of the any function is a number...

Solution 2:

I got the same error message when using sklearn with pandas. My solution is to reset the index of my dataframe df before running any sklearn code:

df = df.reset_index()

I encountered this issue many times when I removed some entries in my df, such as

df = df[df.label=='desired_one']

Solution 3:

When using the Affinity Propagation algorithm from the scikit-learn library, encountering infinite values in your input matrix can lead to errors during computation. Here’s a detailed guide on how to identify and handle these infinite values, along with code examples.

### Step 1: Identify Infinite Values

First, you need to check if your input matrix contains any infinite values. You can do this using NumPy functions.

python
import numpy as np
# Example input matrix
input_matrix = np.array([[1, 2, 3],
 [4, np.inf, 6],
 [7, 8, 9]])
# Check for infinite values
if np.isinf(input_matrix).any():
 print("Input matrix contains infinite values.")
else:
 print("Input matrix does not contain infinite values.")

### Step 2: Handle Infinite Values

If your input matrix contains infinite values, you can replace them with a large finite number or remove the rows/columns containing them. Here are two approaches:

#### Method 1: Replace Infinite Values

You can replace infinite values with a large number (e.g., the maximum value in the matrix):

python
# Replace infinite values with a large number
input_matrix[np.isinf(input_matrix)] = np.finfo(np.float64).max
print("Matrix after replacing infinite values:")
print(input_matrix)

#### Method 2: Remove Rows/Columns with Infinite Values

Alternatively, you can remove rows or columns that contain infinite values:

python
# Remove rows with infinite values
input_matrix_cleaned = input_matrix[~np.isinf(input_matrix).any(axis=1)]
print("Matrix after removing rows with infinite values:")
print(input_matrix_cleaned)

### Step 3: Use Affinity Propagation

Now that the infinite values are handled, you can proceed to use the Affinity Propagation algorithm:

python
from sklearn.cluster import AffinityPropagation
# Use the cleaned input matrix
af = AffinityPropagation()
af.fit(input_matrix_cleaned)
# Get cluster centers and labels
cluster_centers_indices = af.cluster_centers_indices_
labels = af.labels_
print("Cluster centers indices:", cluster_centers_indices)
print("Labels for each point:", labels)

### Example Output

Assuming we replaced the infinite values, the output would look something like this:


Input matrix contains infinite values.
Matrix after replacing infinite values:
[[1.00000000e+00 2.00000000e+00 3.00000000e+00]
 [4.00000000e+00 1.79769313e+308 6.00000000e+00]
 [7.00000000e+00 8.00000000e+00 9.00000000e+00]]
Cluster centers indices: [0 2]
Labels for each point: [0 1 0]

### Conclusion

By identifying and handling infinite values in your input matrix, you can successfully run the Affinity Propagation algorithm. Remember to choose the method that best fits your data and analysis needs. If you continue to face issues, ensure that your data preprocessing steps are robust and check for other potential sources of errors.

Solution 4:

This is my function (based on this) to clean the dataset of nan, Inf, and missing cells (for skewed datasets):

import pandas as pdimport numpy as npdef clean_dataset(df): assert isinstance(df, pd.DataFrame), "df needs to be a pd.DataFrame" df.dropna(inplace=True) indices_to_keep = ~df.isin([np.nan, np.inf, -np.inf]).any(axis=1) return df[indices_to_keep].astype(np.float64)

Solution 5:

The error you're encountering with affinity propagation in sklearn is likely due to the presence of infinite values in your input matrix. To remove these infinite values, you can use the numpy.nan_to_num() function, which replaces infinite values with the largest finite value.

Here's an example of how you can use this function to preprocess your input matrix:

python
import numpy as np
from sklearn.cluster import AffinityPropagation
# Assume your input matrix is stored in the variable input_matrix
input_matrix = np.array([[1, 2, np.inf], [3, np.inf, 5], [np.inf, 6, 7]])
# Remove infinite values using numpy's nan_to_num function
processed_matrix = np.nan_to_num(input_matrix)
# Create an AffinityPropagation object and fit it to the processed matrix
ap = AffinityPropagation(preference=-50).fit(processed_matrix)
# View the cluster centers
print(ap.cluster_centers_indices_)

In this example, the np.inf values in input_matrix are replaced with the largest finite value using np.nan_to_num(), and the resulting processed_matrix can be used with AffinityPropagation without encountering errors.

Note that the value of the preference parameter passed to AffinityPropagation is arbitrary in this example, and you should choose a value that is appropriate for your specific use case. Also note that if your input matrix contains negative infinities, np.nan_to_num() will replace them with negative largest finite value.

Also, make sure that the input matrix is symmetric, if not you can use np.matrix(input_matrix) = np.matrix(input_matrix).T + np.matrix(input_matrix) to symmetrize the matrix before removing infinite values.

I hope this helps you resolve the issue you're encountering with affinity propagation in sklearn!

Solution 6:

The error message you are encountering is caused by infinite values in your input matrix. The affinity propagation algorithm cannot handle infinite values, so you will need to remove them before using the algorithm.

There are a few different ways to remove infinite values from a matrix. One way is to use the numpy.isfinite() function, which returns a boolean mask indicating which elements of the matrix are finite. You can then use this mask to filter out the infinite values.