---
title: "Non-Negative Least Squares (NNLS)"
author: "gobalkrishnan.books@gmail.com"
format: revealjs
fontsize: 22pt
---

# Overview

- **Introduction**: Non-Negative Least Squares (NNLS) is an optimization technique used to find solutions to linear systems where the solutions must be non-negative. This method is crucial in various fields, including economics and signal processing, where negative values do not make sense.

# Definition

- NNLS aims to minimize the sum of the squared differences between observed values and values predicted by a linear model, with the constraint that the coefficients of the model remain non-negative.

# Formula

- **Objective Function**:  
  $$
  \min \| Ax - b \|^2 
  $$ 
  subject to  
  
  $$
  x \geq 0 
  $$  

  where:

  - $A$ is the matrix of predictors.

  - $x$ is the vector of coefficients to be 
  determined.

  - $b$ is the vector of observed values.

# Explanation of Parameters

- **$A$**: A matrix of input features or predictors.
- **$x$**: A vector of coefficients that NNLS aims to estimate, constrained to be non-negative.
- **$b$**: A vector of observed target values or outcomes.

# Advantages

  - Ensures that all coefficients are non-negative, which is essential in many real-world applications.
  - Provides robust solutions that are less sensitive to data anomalies.
  - Applicable in various fields, including finance, engineering, and machine learning.

# Properties

  - The solution is unique if the matrix $A$ has full column rank.
  - Efficient algorithms exist for solving NNLS problems, making it suitable for large datasets.
  - Non-negativity constraints improve interpretability in applications like resource allocation.

# Pros and Cons

| Pros                                   | Cons                                      |
|----------------------------------------|-------------------------------------------|
| Guarantees non-negative solutions      | May not find the optimal solution for all datasets |
| Robust against anomalies                | Can be computationally intensive for very large datasets |
| Provides interpretable results         | Limited flexibility in modeling compared to unrestricted least squares |

# Step-by-Step Calculation

1. **Step 1**: Formulate the optimization problem, defining the objective function and constraints.
2. **Step 2**: Set up the matrix $A$ and vector $b$ using the available data.
3. **Step 3**: Apply an appropriate NNLS algorithm (like active-set or quadratic programming) to find the solution $x$.
4. **Step 4**: Validate the solution to ensure all coefficients are non-negative.

# Example Problem

- **Problem Statement**: A company wants to allocate its budget among three departments (A, B, and C) while ensuring that the total expenditure does not exceed available funds. The observed outcomes based on the previous allocations need to be fitted to understand the best non-negative allocations.

# Given Data

| Department | Predicted Outcome | Expenditure Limit |
|------------|------------------|-------------------|
| A          | 4                | 10                |
| B          | 3                | 15                |
| C          | 5                | 12                |

# Calculation Steps

1. **Step 1**: Construct matrix $A$ representing the relationships and the vector $b$ with the outcomes.
2. **Step 2**: Formulate the NNLS optimization problem.
3. **Step 3**: Solve for $x$ using an NNLS algorithm.


# Solution

1. **Step 1**: Using a solver (like `scipy.optimize.nnls`), find the optimal budget allocation $x$.
2. **Step 2**: Confirm that all elements of $x$ are non-negative.
3. **Step 3**: Present the final allocations for each department.

# Python Code Example using Formula

1. **Step 1**: Setup the problem using NumPy and SciPy.
2. **Step 2**: Utilize the `nnls` function for calculations.


```python
import numpy as np
from scipy.optimize import nnls

# Given data
A = np.array([[1, 1, 1], [0, 1, 1], [1, 0, 1]])  # Example matrix
b = np.array([10, 15, 12])  # Outcome vector

# Solve NNLS
x, rnorm = nnls(A, b)

print("Non-Negative Coefficients (Budget Allocations):", x)
```



```{python}
import numpy as np
from scipy.optimize import nnls

# Given data
A = np.array([[1, 1, 1], [0, 1, 1], [1, 0, 1]])  # Example matrix
b = np.array([10, 15, 12])  # Outcome vector

# Solve NNLS
x, rnorm = nnls(A, b)

print("Non-Negative Coefficients (Budget Allocations):", x)
```

# Python Code Example using Scikit

1. **Step 1**: Import libraries.
2. **Step 2**: Create a dataset.
3. **Step 3**: Fit a model with non-negativity constraints.
4. **Step 4**: Train the model.
5. **Step 5**: Predict outcomes.
6. **Step 6**: Evaluate and visualize results.

---

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression

# Given data
A = np.array([[1, 1, 1], [0, 1, 1], [1, 0, 1]])  # Input features
b = np.array([10, 15, 12])  # Target values

# Fit model with non-negativity constraint
model = LinearRegression(positive=True).fit(A, b)

# Predictions
predictions = model.predict(A)

# Plot results
plt.scatter(b, predictions, label='Predicted vs Observed')
plt.plot(b, b, color='red', linestyle='--', label='Ideal Fit')
plt.xlabel('Observed Values')
plt.ylabel('Predicted Values')
plt.title('NNLS Predictions')
plt.legend()
plt.show()
```
---

```{python}
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression

# Given data
A = np.array([[1, 1, 1], [0, 1, 1], [1, 0, 1]])  # Input features
b = np.array([10, 15, 12])  # Target values

# Fit model with non-negativity constraint
model = LinearRegression(positive=True).fit(A, b)

# Predictions
predictions = model.predict(A)

# Plot results
plt.scatter(b, predictions, label='Predicted vs Observed')
plt.plot(b, b, color='red', linestyle='--', label='Ideal Fit')
plt.xlabel('Observed Values')
plt.ylabel('Predicted Values')
plt.title('NNLS Predictions')
plt.legend()
plt.show()
```

# Applications

- **Applications**:
  - Budget allocation in finance and economics.
  - Signal reconstruction in communication systems.
  - Data fitting in various scientific and engineering applications.

# Summary

- **Summary**: Non-Negative Least Squares is a powerful optimization technique that ensures non-negative solutions for linear models, making it applicable in numerous real-world scenarios.

# Recap

- **Recap**: NNLS helps in solving linear systems while enforcing non-negativity, providing clear and interpretable results, essential in fields where negative values are not logical.