Thought about improving model accuracy for a couple of seconds

Improving a time-series model’s accuracy typically involves multiple dimensions, including data preprocessing, model architecture, hyperparameter tuning, and training strategy. Below are several suggestions you can try. You might not need to do all of them—start with small changes and see which ones most effectively improve performance.

1. Data-Related Improvements

1.1. Check data quality

• Outliers and anomalies: Outliers can harm the training process if they dominate the loss function. Consider removing or clipping extreme values.
• Missing values: Ensure there are no missing values or that they’re properly imputed before training.
• Seasonality or trends: If your data has strong trends or seasonal patterns, consider using techniques such as detrending or adding time-based features (e.g., day of week, month, etc.).

1.2. Try different scalers

• The MinMaxScaler forces data into [0, 1]. However, if your data has outliers or long tails, try StandardScaler or RobustScaler and see if it affects performance.

1.3. Increase the number of features (if available)

• If there are other relevant features (e.g., external factors, weather, holiday flags, etc.), include them in your model. More context sometimes helps the model learn better.

2. Model Architecture and Hyperparameters

2.1. Increase model complexity

• Hidden size: Increase hidden_size from 32 to 64 or 128.
• Number of layers: Use a stacked LSTM (e.g., num_layers=2 or 3).

However, be mindful: too large a model may overfit if you have limited data. Use a validation set or cross-validation to monitor overfitting.

2.2. Add regularization

• Dropout: Apply dropout on the LSTM. For instance:

  self.lstm = nn.LSTM(input_size, hidden_size, num_layers, 
                      batch_first=True, dropout=0.2)
  

  (Note that if num_layers=1, PyTorch’s LSTM ignores the dropout parameter, so consider num_layers > 1 if you want dropout to be active within the LSTM layers.)

• Weight decay: Use a small weight decay in your optimizer to reduce overfitting, for example:

  optimizer = optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-5)
  

2.3. Tune the learning rate

• The default lr=1e-3 may be too high or too low. Try different learning rates (e.g., 1e-2, 5e-4, 1e-4) and see what yields the best validation loss.

2.4. Increase number of epochs

• If your model is still improving at the end of 30 epochs, increase to 50 or 100. Plot training vs. validation loss to see if there’s a plateau or signs of overfitting.

2.5. Consider alternative architectures

• GRU: Replace the LSTM with a GRU layer. GRUs often train faster and can perform comparably or better in some time-series tasks.
• Temporal Convolution: In some cases, a 1D Conv (e.g., a Temporal Convolutional Network) can outperform RNN-based models on time-series data.

3. Training Strategy

3.1. Use a validation set

• Split your data into three parts: train, validation, and test. Use the validation set to tune hyperparameters (learning rate, hidden size, etc.). Only check test set performance at the end.

3.2. Early stopping

• Monitor validation loss and stop training once it stops improving (or starts deteriorating). This prevents overfitting and might save you time.

3.3. Batch size

• Experiment with different batch sizes. A batch size of 32 might work, but 16 or 64 could yield better results.

4. Accuracy Metric and Threshold

4.1. Rethink the “accuracy” definition

Right now, you’re defining accuracy in the scaled space as:

epsilon = 0.1
diff = np.linalg.norm(predictions - y_test_np, axis=1)
accuracy = np.mean(diff < epsilon)

A norm difference less than 0.1 on a 2D scaled vector might be quite strict or too lenient, depending on the scale of your data:

1. Tune epsilon: If your data has certain variance, epsilon=0.1 might be too small. You can experiment with different thresholds, e.g., 0.05, 0.2, etc.
2. Evaluate dimension-wise: Instead of using the vector norm, you might also consider checking each feature’s absolute difference. For example:

   accuracy_feature1 = np.mean(np.abs(predictions[:, 0] - y_test_np[:, 0]) < epsilon_feature1)
   accuracy_feature2 = np.mean(np.abs(predictions[:, 1] - y_test_np[:, 1]) < epsilon_feature2)
   

   Then average these or interpret them individually.

4.2. Stick to continuous error metrics

• For time-series, it’s often more typical to rely on RMSE, MAE, or MAPE (mean absolute percentage error). The notion of “accuracy” might be less intuitive unless you have a clear classification-like criterion.

5. Putting it All Together (Example Changes)

Below is an example of how you might modify your code in small steps.

1. Increase hidden size and layers, and add dropout:

   class LSTMModel(nn.Module):
       def __init__(self, input_size=2, hidden_size=64, num_layers=2, output_size=2, dropout=0.2):
           super(LSTMModel, self).__init__()
           self.hidden_size = hidden_size
           self.num_layers = num_layers
           self.lstm = nn.LSTM(input_size, hidden_size, num_layers=num_layers,
                               batch_first=True, dropout=dropout)
           self.fc = nn.Linear(hidden_size, output_size)
       
       def forward(self, x):
           # Initialize hidden states
           h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size)
           c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size)
           
           out, _ = self.lstm(x, (h0, c0))    # out shape: (batch, seq_len, hidden_size)
           out = out[:, -1, :]               # only last time step
           out = self.fc(out)
           return out
   

2. Use weight decay and possibly tweak lr:

   optimizer = optim.Adam(model.parameters(), lr=5e-4, weight_decay=1e-5)
   

3. Train for more epochs or use early stopping:

   best_val_loss = float('inf')
   for epoch in range(100):
       model.train()
       for X_batch, y_batch in train_loader:
           optimizer.zero_grad()
           outputs = model(X_batch)
           loss = criterion(outputs, y_batch)
           loss.backward()
           optimizer.step()
       
       # Validation step (if you have a validation set)
       model.eval()
       with torch.no_grad():
           val_outputs = model(X_val_tensors)
           val_loss = criterion(val_outputs, y_val_tensors).item()
       
       # Early stopping check
       if val_loss < best_val_loss:
           best_val_loss = val_loss
           # Save best model if you like
       else:
           # If val_loss didn't improve, consider stopping or adjusting
           pass
       
       # Print progress
       if (epoch + 1) % 10 == 0:
           print(f"Epoch [{epoch+1}/100], Train Loss: {loss.item():.6f}, Val Loss: {val_loss:.6f}")
   

4. Consider standard scaling:

   from sklearn.preprocessing import StandardScaler
   
   scaler = StandardScaler()
   scaled_data = scaler.fit_transform(df[['col1', 'col2']])
   # Keep the rest of your code the same
   

5. Adjust your “accuracy” threshold or remove it altogether** and rely on RMSE/MAE if that is more relevant for your problem.

Summary

• Data: Explore new features, remove outliers, and carefully choose a scaler.
• Architecture: Use stacked LSTM or GRU with a larger hidden size, dropout, and possibly weight decay.
• Hyperparameters: Tune the learning rate, batch size, and number of epochs. Consider early stopping.
• Metrics: Choose thresholds carefully if using accuracy in a regression setting. RMSE, MAE, or MAPE are often more informative for time-series.

Incrementally applying these tips should help you improve your model’s performance and push that 17% accuracy closer to a more acceptable range.

Integrating components

I’m pulling together components like an LSTM model, GridSearch, early stopping, and a custom accuracy metric into a unified PyTorch training and evaluation script.

Crafting the function

Working on the train_and_evaluate function to manually handle hyperparameter optimization, integrating early stopping. Addresses model creation, data loading, and defines hyperparameters like hidden_size, num_layers, and lr.

Strategizing data division

I’m piecing together a data split—70% training, 10% validation, and 20% test. This clear segmentation guides hyperparameter tuning and model training, aiming for a well-rounded evaluation.

Pulling together

I’m integrating functions for calculating accuracy and staging data. This includes the LSTM model, hidden_size, num_layers, and the dataset.

Adjusting data handling

OK, I’m tweaking the user's data approach to a 70/10/20 train-validation-test split, balancing validation set use for early stopping with the original 80/20 train-test split.

Thought about LSTM tuning with grid search and early stopping for 15 seconds

Below is a complete code example showing how you could incorporate grid search for hyperparameter tuning and early stopping into your current workflow. In this example, we do the following:

1. Split the data into train, validation, and test sets (70/10/20).
2. Define a grid of hyperparameters (e.g., hidden sizes, number of layers, learning rates).
3. Implement early stopping during training by monitoring validation loss.
4. Use your custom accuracy function (calculate_accuracy) with a 10% threshold.
5. At the end of the grid search, we pick the best hyperparameters based on validation performance and evaluate on the final test set.

Feel free to modify the splits or the hyperparameter search space to fit your exact needs.

import pandas as pd
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.preprocessing import MinMaxScaler
from torch.utils.data import TensorDataset, DataLoader
import matplotlib.pyplot as plt

############################################
# 1. Load data
############################################
df = pd.read_csv("realDataForTrain.csv")

############################################
# 2. Normalize
############################################
scaler = MinMaxScaler()
scaled_data = scaler.fit_transform(df[['col1', 'col2']])

############################################
# 3. Create sequences
############################################
SEQ_LENGTH = 10

def create_sequences(data, seq_length=10):
    X, y = [], []
    for i in range(len(data) - seq_length):
        X.append(data[i : i + seq_length])
        y.append(data[i + seq_length])
    return np.array(X), np.array(y)

X, y = create_sequences(scaled_data, SEQ_LENGTH)

# Let's do 70% train, 10% validation, 20% test
train_size = int(len(X) * 0.7)
val_size   = int(len(X) * 0.1)
test_size  = len(X) - train_size - val_size

# Indices
X_train = X[:train_size]
y_train = y[:train_size]

X_val   = X[train_size : train_size + val_size]
y_val   = y[train_size : train_size + val_size]

X_test  = X[train_size + val_size:]
y_test  = y[train_size + val_size:]

# Convert to tensors
X_train_tensors = torch.tensor(X_train, dtype=torch.float32)
y_train_tensors = torch.tensor(y_train, dtype=torch.float32)

X_val_tensors   = torch.tensor(X_val,   dtype=torch.float32)
y_val_tensors   = torch.tensor(y_val,   dtype=torch.float32)

X_test_tensors  = torch.tensor(X_test,  dtype=torch.float32)
y_test_tensors  = torch.tensor(y_test,  dtype=torch.float32)

############################################
# 4. Define LSTM model
############################################
class LSTMModel(nn.Module):
    def __init__(self, input_size=2, hidden_size=32, num_layers=1, output_size=2):
        super(LSTMModel, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)
        
    def forward(self, x):
        # Initialize hidden state and cell state
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size)
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size)
        
        out, _ = self.lstm(x, (h0, c0))   # out: (batch, seq_length, hidden_size)
        out = out[:, -1, :]              # Take the last time step
        out = self.fc(out)               # Linear layer
        return out

############################################
# 5. Accuracy calculation (10% threshold)
############################################
def calculate_accuracy(y_true, y_pred, threshold=0.10):
    """
    Calculate accuracy based on how many predictions are 
    within a given percentage threshold of the true values.
    """
    # y_true and y_pred shape: (batch, 2)
    within_tolerance = np.abs(y_true - y_pred) <= threshold * np.abs(y_true)
    # within_tolerance is a boolean array of shape (batch, 2)
    accuracy = np.mean(within_tolerance) * 100  # Convert to percentage
    return accuracy

############################################
# 6. Training routine with early stopping
############################################
def train_model(model, 
                train_loader, 
                val_data, 
                criterion, 
                optimizer, 
                max_epochs=100, 
                patience=10):
    """
    Train the model with early stopping on validation loss.
    Args:
        model: The PyTorch model
        train_loader: DataLoader for training
        val_data: (X_val_tensors, y_val_tensors)
        criterion: Loss function
        optimizer: Optimizer
        max_epochs: Maximum number of epochs
        patience: How many epochs to wait for improvement before stopping
    Returns:
        best_model: The best model (state) found
        best_val_loss: The validation loss of the best model
        history: A dict with training/validation loss for plotting
    """
    X_val_tensors, y_val_tensors = val_data
    best_val_loss = float('inf')
    epochs_no_improve = 0
    history = {'train_loss': [], 'val_loss': []}
    
    for epoch in range(max_epochs):
        # Train mode
        model.train()
        train_loss_sum = 0.0
        
        for X_batch, y_batch in train_loader:
            optimizer.zero_grad()
            outputs = model(X_batch)
            loss = criterion(outputs, y_batch)
            loss.backward()
            optimizer.step()
            train_loss_sum += loss.item()
        
        train_loss = train_loss_sum / len(train_loader)
        
        # Validation
        model.eval()
        with torch.no_grad():
            val_outputs = model(X_val_tensors)
            val_loss = criterion(val_outputs, y_val_tensors).item()
        
        history['train_loss'].append(train_loss)
        history['val_loss'].append(val_loss)
        
        # Check for improvement
        if val_loss < best_val_loss:
            best_val_loss = val_loss
            best_model_state = model.state_dict()  # save model state
            epochs_no_improve = 0
        else:
            epochs_no_improve += 1
        
        # Early stopping
        if epochs_no_improve >= patience:
            print(f"Early stopping on epoch {epoch+1}")
            break
        
        if (epoch + 1) % 5 == 0:
            print(f"Epoch [{epoch+1}/{max_epochs}] "
                  f"Train Loss: {train_loss:.6f}, Val Loss: {val_loss:.6f}")
    
    # Load the best model state
    model.load_state_dict(best_model_state)
    return model, best_val_loss, history

############################################
# 7. Grid search
############################################
from torch.utils.data import TensorDataset, DataLoader

# Prepare train DataLoader
train_dataset = TensorDataset(X_train_tensors, y_train_tensors)

# Hyperparameter grid
hidden_sizes = [32, 64]
num_layers_list = [1, 2]
learning_rates = [1e-3, 1e-4]
batch_sizes = [16, 32]

best_hparams = None
best_val_loss = float('inf')
best_model = None

# We will do a simple manual grid search
for hidden_size in hidden_sizes:
    for num_layers in num_layers_list:
        for lr in learning_rates:
            for batch_size in batch_sizes:
                print(f"\nTrying: hidden_size={hidden_size}, num_layers={num_layers}, lr={lr}, batch_size={batch_size}")
                
                # Create model
                model = LSTMModel(input_size=2, 
                                  hidden_size=hidden_size, 
                                  num_layers=num_layers, 
                                  output_size=2)
                criterion = nn.MSELoss()
                optimizer = optim.Adam(model.parameters(), lr=lr)
                
                # Create DataLoader with the chosen batch_size
                train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
                
                # Train with early stopping
                model, val_loss, history = train_model(
                    model=model,
                    train_loader=train_loader,
                    val_data=(X_val_tensors, y_val_tensors),
                    criterion=criterion,
                    optimizer=optimizer,
                    max_epochs=50,     # you can increase if you want
                    patience=5         # early stopping patience
                )
                
                print(f"Finished. Val Loss = {val_loss:.6f}")
                
                # Check if this is the best so far
                if val_loss < best_val_loss:
                    best_val_loss = val_loss
                    best_hparams = (hidden_size, num_layers, lr, batch_size)
                    # Clone the model to keep it
                    best_model = LSTMModel(input_size=2, 
                                           hidden_size=hidden_size, 
                                           num_layers=num_layers, 
                                           output_size=2)
                    best_model.load_state_dict(model.state_dict())
                    print("==> New best hyperparameters found.")

print(f"\nBest val loss = {best_val_loss:.6f} with hparams: hidden_size={best_hparams[0]}, "
      f"num_layers={best_hparams[1]}, lr={best_hparams[2]}, batch_size={best_hparams[3]}")

############################################
# 8. Final Evaluation on Test Set
############################################
best_model.eval()
with torch.no_grad():
    # Predictions in scaled space
    predictions = best_model(X_test_tensors).numpy()

y_test_np = y_test_tensors.numpy()
# Inverse transform predictions and ground truth to original scale
predictions_inv = scaler.inverse_transform(predictions)
y_test_inv = scaler.inverse_transform(y_test_np)

# Calculate standard regression metrics in scaled space
mse = np.mean((predictions - y_test_np) ** 2)
rmse = np.sqrt(mse)
mae = np.mean(np.abs(predictions - y_test_np))

# R^2 in scaled space
ss_res = np.sum((y_test_np - predictions) ** 2)
ss_tot = np.sum((y_test_np - np.mean(y_test_np, axis=0)) ** 2)
r2 = 1 - (ss_res / ss_tot)

print("\n=== Test Set Performance (Scaled Space) ===")
print(f"MSE : {mse:.6f}")
print(f"RMSE: {rmse:.6f}")
print(f"MAE : {mae:.6f}")
print(f"R^2 : {r2:.6f}")

# Calculate custom accuracy in scaled space (element-wise within 10% of y_true)
accuracy_10pct = calculate_accuracy(y_test_np, predictions, threshold=0.10)
print(f"Accuracy (10% threshold in scaled space): {accuracy_10pct:.2f}%")

############################################
# 9. Predict the next 10 values
############################################
def predict_future(model, data, scaler, seq_length=10, steps=10):
    model.eval()
    window = data[-seq_length:].copy()
    preds = []
    for _ in range(steps):
        x_t = torch.tensor(window[np.newaxis, :, :], dtype=torch.float32)
        with torch.no_grad():
            pred = model(x_t)
        pred_np = pred.numpy()[0]
        preds.append(pred_np)
        # Update the "window"
        window = np.vstack([window[1:], pred_np])
    preds = np.array(preds)
    return scaler.inverse_transform(preds)

future_preds = predict_future(best_model, scaled_data, scaler, seq_length=SEQ_LENGTH, steps=10)
print("\nNext 10 predictions (col1, col2):")
for i, (p1, p2) in enumerate(future_preds, start=1):
    print(f"Step {i}: col1={p1:.4f}, col2={p2:.4f}")

############################################
# 10. (Optional) Plot
############################################
plt.figure(figsize=(12, 4))

test_index = range(train_size + val_size + SEQ_LENGTH, train_size + val_size + SEQ_LENGTH + len(y_test_inv))
plt.plot(test_index, y_test_inv[:, 0], label="Actual col1", color="blue")
plt.plot(test_index, predictions_inv[:, 0], label="Predicted col1", color="red")
plt.title("col1 Actual vs Predicted (Test Set)")
plt.legend()
plt.show()

Key Points to Note

1. Train/Val/Test Split
   We changed the split to 70/10/20, so we’d have a dedicated validation set for early stopping and hyperparameter tuning. If you want to keep your 80/20 train/test split exactly, you can carve the validation set out of the training split.

2. Grid Search

   • We manually iterate through various hidden sizes, number of layers, learning rates, and batch sizes.
   • For each combination, we train a new model from scratch and keep track of the best validation loss.
   • You can easily expand or change these ranges (e.g., add dropout or different optimizers).

3. Early Stopping

   • We set a patience parameter that stops training if the validation loss does not improve for patience consecutive epochs.
   • The best model parameters (lowest validation loss) are stored and then reloaded at the end of training.

4. Accuracy Function (calculate_accuracy)

   • This uses your 10% threshold formula on the scaled predictions. It checks element-wise within 10% for both col1 and col2.
   • We convert the final mean to a percentage.

5. Final Model on Test Set

   • After grid search picks the best hyperparameters, we evaluate that best model on the test set and print out MSE, RMSE, MAE, R^2, and your accuracy metric.

6. Predict Future

   • The predict_future function is unchanged from your original code, but here we apply it to the best model found from grid search.

Feel free to adjust any details (e.g., patience, max_epochs, or hyperparameter grid) to suit your data and to find the best performance.

Advancing strategies

I’m working through sophisticated data cleaning, scaling, feature engineering, and model architecture. Considering an accuracy metric and exploring ensemble methods for enhanced performance.

Refining the approach

I'm addressing the need for refining scaling, considering separate models for columns, and exploring dropout and architecture changes. Negative R^2 highlights the model's underperformance.

Thought about improving accuracy for 6 seconds

Below is a comprehensive list of suggestions and code modifications you can try in order to significantly boost your accuracy (with the 10% threshold definition). The main takeaways will be:

1. Data preparation (potentially the biggest factor).
2. Model architecture (stacked LSTM, larger hidden dimension, dropout).
3. Hyperparameter tuning (including learning rate, batch size, epochs).
4. Rethinking the threshold-based accuracy or ensuring it’s appropriate for your data distribution.

Important: Achieving an 80%+ accuracy with the threshold-based metric can be challenging depending on the data’s volatility, magnitude differences, etc. If your data varies greatly (e.g., col2 ranges up to 100, while col1 is around ±1 or so), you may need to treat each column differently (e.g., scale or transform each column separately, or compute accuracy per column). Nonetheless, let’s walk through the main steps.

1. Data-Related Enhancements

1. Separate Scaling for Each Column

   • If col2 is on a much different scale than col1, using the same MinMaxScaler for both might distort one relative to the other. Instead, consider:

     scaler_col1 = MinMaxScaler()
     scaler_col2 = MinMaxScaler()
     data_col1 = df[['col1']].values
     data_col2 = df[['col2']].values
     
     data_col1_scaled = scaler_col1.fit_transform(data_col1)
     data_col2_scaled = scaler_col2.fit_transform(data_col2)
     
     # Combine them back or keep them separate
     scaled_data = np.hstack([data_col1_scaled, data_col2_scaled])
     

   • Then, when inversely transforming, you must inverse each column separately.

2. Check Outliers

   • If you have large negative or positive spikes in col1 or col2, the LSTM might “overfit” to those points. Consider removing or clipping extreme outliers if they are not valid data points.

3. Temporal/Seasonal Features

   • If your data is time-indexed (e.g., daily, hourly), consider adding time-based features (like day of week, month, etc.) or additional relevant exogenous variables that may help the model learn patterns.

4. Smoothing

   • For some tasks, smoothing the output (e.g., a rolling average) can improve predictive performance if you only need approximate trends. But be careful if your application requires precise short-term predictions.

2. Model Architecture

To get higher accuracy, especially if your dataset is large or has complex patterns:

1. Use a Larger Hidden Size

   • For example, try hidden_size=64 or 128.

2. Use Multiple LSTM Layers (Stacked LSTM)

   • num_layers=2 or 3 can capture more complex temporal dependencies.

3. Add Dropout

   • E.g.:

     self.lstm = nn.LSTM(input_size, hidden_size, num_layers, 
                         batch_first=True, dropout=0.2)
     

   • Note that dropout only works if num_layers > 1 in PyTorch’s LSTM.

4. Try GRU

   • A GRU can sometimes train faster and yield better performance. Simply replace nn.LSTM with nn.GRU (with minimal code changes).

3. Training Strategy

1. Increase Number of Epochs
   • If your model is still improving after 50 epochs, train longer (e.g., 100, 200).
2. Tune Learning Rate
   • If lr=1e-3 is not working, try 5e-4, 1e-4, or even bigger like 5e-3.
   • You can also try advanced optimizers or learning rate schedules (e.g., ReduceLROnPlateau).
3. Add Weight Decay
   • A small weight decay (e.g. weight_decay=1e-5) can improve generalization.
4. Don’t Shuffle Time Series
   • Make sure your training DataLoader is not incorrectly shuffling time steps that should remain sequential.
   • Typically, you do want to shuffle the batches in a time-series sense (i.e., each batch is a consecutive chunk) or use a rolling window. But do not mix rows from the far past with the far future in the same batch if it breaks the time ordering.

4. Revisit the Accuracy Definition

You’re using:

def calculate_accuracy(y_true, y_pred, threshold=0.10):
    within_tolerance = np.abs(y_true - y_pred) <= threshold * np.abs(y_true)
    accuracy = np.mean(within_tolerance) * 100
    return accuracy

Tips:

• If y_true values are near zero, the term threshold * np.abs(y_true) becomes very small, making it almost impossible to stay within 10%.
• If your data has zeros or very small values, you might need a small offset or a different approach (e.g., (pred - true) / (mean_of_true + 1)).
• Consider measuring accuracy per column and then combining them. For example, you might get 90% accuracy for col1 and 30% for col2 or vice versa. Then you see which column is the bottleneck.

5. Example Code with Enhancements

Below is a refined version of the code you already have with the following changes:

• Separate scaling for col1 and col2.
• Bigger hidden size and stacked LSTM with dropout.
• Early stopping and grid search are retained from the previous example.
• More epochs (up to 100, with patience=10 for early stopping).
• Weight decay added to Adam.
• Accuracy calculation remains the same, but we print accuracy for each column separately, too (to diagnose differences).

You can incrementally tweak these parameters (hidden_size, num_layers, dropout, lr, etc.) and see if you approach or exceed that 80% threshold accuracy.

import pandas as pd
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.preprocessing import MinMaxScaler
from torch.utils.data import TensorDataset, DataLoader
import matplotlib.pyplot as plt

############################################
# 1. Load data
############################################
df = pd.read_csv("realDataForTrain.csv")

############################################
# 2. Separate scaling for col1 and col2
############################################
col1_data = df[['col1']].values  # shape: (N, 1)
col2_data = df[['col2']].values  # shape: (N, 1)

scaler_col1 = MinMaxScaler()
scaler_col2 = MinMaxScaler()

col1_scaled = scaler_col1.fit_transform(col1_data)  # shape (N, 1)
col2_scaled = scaler_col2.fit_transform(col2_data)  # shape (N, 1)

# Combine scaled columns
scaled_data = np.hstack([col1_scaled, col2_scaled])  # shape (N, 2)

############################################
# 3. Create sequences
############################################
SEQ_LENGTH = 10

def create_sequences(data, seq_length=10):
    X, y = [], []
    for i in range(len(data) - seq_length):
        X.append(data[i : i + seq_length])
        y.append(data[i + seq_length])  # next point
    return np.array(X), np.array(y)

X, y = create_sequences(scaled_data, SEQ_LENGTH)

# We do 70% train, 10% val, 20% test
train_size = int(0.7 * len(X))
val_size   = int(0.1 * len(X))
test_size  = len(X) - train_size - val_size

X_train = X[:train_size]
y_train = y[:train_size]

X_val = X[train_size : train_size + val_size]
y_val = y[train_size : train_size + val_size]

X_test = X[train_size + val_size:]
y_test = y[train_size + val_size:]

# Convert to tensors
X_train_tensors = torch.tensor(X_train, dtype=torch.float32)
y_train_tensors = torch.tensor(y_train, dtype=torch.float32)

X_val_tensors = torch.tensor(X_val, dtype=torch.float32)
y_val_tensors = torch.tensor(y_val, dtype=torch.float32)

X_test_tensors = torch.tensor(X_test, dtype=torch.float32)
y_test_tensors = torch.tensor(y_test, dtype=torch.float32)

############################################
# 4. Define LSTM Model (Stacked + Dropout)
############################################
class LSTMModel(nn.Module):
    def __init__(self, input_size=2, hidden_size=64, num_layers=2, output_size=2, dropout=0.2):
        super(LSTMModel, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm = nn.LSTM(
            input_size, hidden_size, 
            num_layers=num_layers, 
            batch_first=True, 
            dropout=dropout
        )
        self.fc = nn.Linear(hidden_size, output_size)
        
    def forward(self, x):
        # Initialize hidden state and cell state with zeros
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size)
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size)
        
        out, _ = self.lstm(x, (h0, c0))   # out: (batch, seq_len, hidden_size)
        out = out[:, -1, :]              # last time step
        out = self.fc(out)               # linear
        return out

############################################
# 5. Accuracy function (10% threshold)
############################################
def calculate_accuracy(y_true, y_pred, threshold=0.10):
    """
    y_true, y_pred: shape (batch, 2)
    We'll calculate fraction of points where
    |y_pred - y_true| <= threshold * |y_true|
    Then convert to percentage
    """
    within_tolerance = np.abs(y_true - y_pred) <= threshold * np.abs(y_true)
    accuracy = np.mean(within_tolerance) * 100
    return accuracy

############################################
# 6. Early Stopping Training Routine
############################################
def train_model(model, train_loader, val_data, criterion, optimizer, 
                max_epochs=100, patience=10):
    X_val_tensors, y_val_tensors = val_data
    best_val_loss = float('inf')
    epochs_no_improve = 0
    
    for epoch in range(max_epochs):
        # Training
        model.train()
        train_loss_sum = 0
        for X_batch, y_batch in train_loader:
            optimizer.zero_grad()
            outputs = model(X_batch)
            loss = criterion(outputs, y_batch)
            loss.backward()
            optimizer.step()
            train_loss_sum += loss.item()
        
        train_loss = train_loss_sum / len(train_loader)
        
        # Validation
        model.eval()
        with torch.no_grad():
            val_outputs = model(X_val_tensors)
            val_loss = criterion(val_outputs, y_val_tensors).item()
        
        if val_loss < best_val_loss:
            best_val_loss = val_loss
            best_model_state = model.state_dict()
            epochs_no_improve = 0
        else:
            epochs_no_improve += 1
        
        if epochs_no_improve >= patience:
            print(f"Early stopping at epoch {epoch+1}")
            break
        
        if (epoch + 1) % 5 == 0:
            print(f"Epoch [{epoch+1}/{max_epochs}] - Train Loss: {train_loss:.6f}, Val Loss: {val_loss:.6f}")
    
    # Load best model
    model.load_state_dict(best_model_state)
    return model, best_val_loss

############################################
# 7. Grid Search (Optional)
############################################
from torch.utils.data import DataLoader

train_dataset = TensorDataset(X_train_tensors, y_train_tensors)

# Let’s do a small grid for demonstration
hidden_sizes = [64, 128]      # bigger hidden sizes
num_layers_list = [1, 2]      # stacked or not
learning_rates = [1e-3, 1e-4]
batch_sizes = [32, 64]
dropouts = [0.0, 0.2]

best_val_loss = float('inf')
best_model = None
best_hparams = None

for hs in hidden_sizes:
    for nl in num_layers_list:
        for lr in learning_rates:
            for bs in batch_sizes:
                for dp in dropouts:
                    print(f"\nTesting hyperparams: hs={hs}, nl={nl}, lr={lr}, bs={bs}, dropout={dp}")
                    
                    # Create model
                    model = LSTMModel(input_size=2, hidden_size=hs, num_layers=nl, dropout=dp)
                    criterion = nn.MSELoss()
                    optimizer = optim.Adam(model.parameters(), lr=lr, weight_decay=1e-5)
                    
                    # Create DataLoader
                    train_loader = DataLoader(train_dataset, batch_size=bs, shuffle=True)
                    
                    # Train
                    model, val_loss = train_model(
                        model=model,
                        train_loader=train_loader,
                        val_data=(X_val_tensors, y_val_tensors),
                        criterion=criterion,
                        optimizer=optimizer,
                        max_epochs=100,  # can increase
                        patience=10
                    )
                    
                    print(f"Val Loss: {val_loss:.6f}")
                    if val_loss < best_val_loss:
                        best_val_loss = val_loss
                        best_hparams = (hs, nl, lr, bs, dp)
                        # Save best model
                        best_model = LSTMModel(input_size=2, hidden_size=hs, num_layers=nl, dropout=dp)
                        best_model.load_state_dict(model.state_dict())
                        print("  ==> New best found!")

print(f"\nBest val loss = {best_val_loss:.6f} with hparams: {best_hparams}")

############################################
# 8. Evaluate on Test
############################################
best_model.eval()
with torch.no_grad():
    test_preds_scaled = best_model(X_test_tensors).numpy()  # shape (test_samples, 2)

y_test_np = y_test_tensors.numpy()  # shape (test_samples, 2)

# Inverse transform each column separately
# test_preds_scaled[:, 0] -> col1, test_preds_scaled[:, 1] -> col2
test_preds_col1 = scaler_col1.inverse_transform(test_preds_scaled[:, [0]])
test_preds_col2 = scaler_col2.inverse_transform(test_preds_scaled[:, [1]])
test_preds_inv = np.hstack([test_preds_col1, test_preds_col2])

# Same for y_test_np
y_test_col1 = scaler_col1.inverse_transform(y_test_np[:, [0]])
y_test_col2 = scaler_col2.inverse_transform(y_test_np[:, [1]])
y_test_inv = np.hstack([y_test_col1, y_test_col2])

# 8.1 - MSE, RMSE, MAE in *scaled* space
mse_scaled = np.mean((test_preds_scaled - y_test_np)**2)
rmse_scaled = np.sqrt(mse_scaled)
mae_scaled = np.mean(np.abs(test_preds_scaled - y_test_np))

# 8.2 - R^2 in scaled space
ss_res = np.sum((y_test_np - test_preds_scaled)**2)
ss_tot = np.sum((y_test_np - np.mean(y_test_np, axis=0))**2)
r2_scaled = 1 - (ss_res / ss_tot)

print("\n=== Test Set Performance (Scaled Space) ===")
print(f"MSE : {mse_scaled:.6f}")
print(f"RMSE: {rmse_scaled:.6f}")
print(f"MAE : {mae_scaled:.6f}")
print(f"R^2 : {r2_scaled:.6f}")

# 8.3 - Accuracy in scaled space (10% threshold)
acc_10pct = calculate_accuracy(y_test_np, test_preds_scaled, threshold=0.10)
print(f"Accuracy (10% threshold, scaled space, both columns): {acc_10pct:.2f}%")

# 8.4 - (Optional) Accuracy for each column separately
col1_acc_10pct = calculate_accuracy(y_test_np[:, [0]], test_preds_scaled[:, [0]], threshold=0.10)
col2_acc_10pct = calculate_accuracy(y_test_np[:, [1]], test_preds_scaled[:, [1]], threshold=0.10)
print(f"Accuracy col1 (10% threshold): {col1_acc_10pct:.2f}%")
print(f"Accuracy col2 (10% threshold): {col2_acc_10pct:.2f}%")

############################################
# 9. Predict the next 10 values
############################################
def predict_future(model, data_scaled_col1, data_scaled_col2,
                   scaler_col1, scaler_col2,
                   seq_length=10, steps=10):
    """
    data_scaled_col1, data_scaled_col2 are 1D arrays of the scaled col1, col2
    We'll stack them last 10 steps to feed the model.
    """
    model.eval()
    
    # Combine them into shape (N, 2)
    combined_scaled = np.hstack([
        data_scaled_col1.reshape(-1,1),
        data_scaled_col2.reshape(-1,1)
    ])
    
    window = combined_scaled[-seq_length:].copy()  # last seq_length rows
    preds = []
    
    for _ in range(steps):
        x_t = torch.tensor(window[np.newaxis, :, :], dtype=torch.float32)
        with torch.no_grad():
            pred = model(x_t).numpy()[0]  # shape (2,)
        preds.append(pred)
        
        # Shift the window
        window = np.vstack([window[1:], pred])
    
    preds = np.array(preds)  # shape (steps, 2)
    
    # Inverse transform each column
    pred_col1_inv = scaler_col1.inverse_transform(preds[:, [0]])
    pred_col2_inv = scaler_col2.inverse_transform(preds[:, [1]])
    preds_inv = np.hstack([pred_col1_inv, pred_col2_inv])
    return preds_inv

future_preds = predict_future(best_model, col1_scaled, col2_scaled,
                              scaler_col1, scaler_col2,
                              seq_length=SEQ_LENGTH, steps=10)

print("\nNext 10 predictions (col1, col2):")
for i, (p1, p2) in enumerate(future_preds, start=1):
    print(f"Step {i}: col1={p1:.4f}, col2={p2:.4f}")

############################################
# 10. Plot
############################################
plt.figure(figsize=(12, 4))
test_index = range(train_size + val_size + SEQ_LENGTH, 
                   train_size + val_size + SEQ_LENGTH + len(y_test_inv))

# Plot col1
plt.plot(test_index, y_test_inv[:, 0], label="Actual col1", color="blue")
plt.plot(test_index, test_preds_inv[:, 0], label="Predicted col1", color="red")
plt.title("col1 Actual vs Predicted (Test Set)")
plt.legend()
plt.show()

6. Further Tips for Getting to 80%+

1. Look at Per-Column Accuracy

   • If col2 has large swings (e.g., from near 0 up to 100), the ±10% band might be too narrow for higher accuracy. Possibly adjust the threshold for each column separately or transform col2 (e.g., log-scale if it’s always positive).

2. Consider a Log or Other Transform

   • If col2 is always positive, you can transform col2 with log1p(col2) (log of col2 + 1), train the model, and then exponentiate predictions. This compresses outliers and might help the model get more consistent predictions.

3. Feature Engineering

   • Add external signals (if relevant) that can help your LSTM see more context about what drives col2.

4. Try More Advanced Architectures

   • E.g., GRU, TCN (Temporal Convolutional Network), or Transformers.
   • Sometimes a simpler architecture (like GRU) might converge faster and generalize better.

5. Check Data Integrity

   • Missing or corrupted data can reduce performance drastically.

6. Hyperparameter Tuning

   • Expand or refine your grid search (or use libraries like Optuna or Ray Tune) to systematically search larger spaces for hidden_size, LR, number of layers, dropout, batch size, and so on.

7. Longer Training

   • If you see your training loss still improving at the end of 100 epochs, you can push further or reduce the learning rate and keep going.

Conclusion

Hitting 80% accuracy (with your threshold-based metric) may require careful data scaling, transformations, a sufficiently powerful (yet well-regularized) model, and thorough hyperparameter tuning. Start by focusing on data (scaling and outliers), and systematically tune the LSTM architecture and training scheme. Good luck!

Mapping out the refactor

I'm integrating the LSTM model for a 10-step, multi-output forecast. This involves enhancing data loading, preparation, and training components to support the refined objectives.

Refining predictions

I'm working on a multi-step forecast for 10 future timesteps across two columns. The output will either be a single 20-dimensional vector or a (batch_size, 10, 2) shape.

Shaping the trainer

I'm refining the training process for multi-step predictions by determining data splitting, accuracy metrics, and adjusting the "predict_future" function to manage sequence prediction dynamically.

Figuring out the solution

I’m moving away from the iterative "predict_future" approach, focusing instead on a simple, efficient code snippet that seamlessly integrates with the provided architecture.

Refining sequences

Crafting "create_sequences_multistep" to produce X and flattened y, training model with specified parameters, and incorporating hyperparameters for consistency.

Thought about refactoring LSTM architecture for 25 seconds

Below is one possible refactoring of your code that uses the 3-layer LSTM architecture you provided and predicts the next 10 time steps for 2 features (thus output_size = 20). This example demonstrates:

• How to create multi-step training sequences.
• How to handle a (forecast_horizon=10)-step-ahead prediction (for both col1 and col2).
• How to shape the targets (y) to match the 20-dimensional output from the final dense layer.
• A simple training loop (no advanced hyperparameter search or early stopping here, but you can add them back if desired).

Adjust any parameters (epochs, batch size, learning rate, etc.) as needed to see better performance.

1) Complete Refactored Code

import pandas as pd
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.preprocessing import MinMaxScaler
from torch.utils.data import TensorDataset, DataLoader
import matplotlib.pyplot as plt

##############################################################################
# 1. Hyperparameters (from your snippet, feel free to tweak as needed)
##############################################################################
input_size = 2       # Two features: (col1, col2)
hidden_size = 64     # 64 hidden units in each LSTM layer
num_layers = 3       # 3-layer LSTM (stacked)
output_size = 20     # Predict 10 time steps * 2 features => 20
dropout = 0.5        # High dropout to reduce overfitting
learning_rate = 0.005
num_epochs = 9
batch_size = 64

##############################################################################
# 2. Load Data
##############################################################################
df = pd.read_csv("realDataForTrain.csv")
# We'll assume your CSV has columns named exactly 'col1' and 'col2'.
# If not, adjust accordingly.
data = df[['col1', 'col2']].values  # shape (N, 2)

##############################################################################
# 3. Scaling
##############################################################################
scaler = MinMaxScaler()
data_scaled = scaler.fit_transform(data)  # shape (N, 2)

##############################################################################
# 4. Create Multi-Step Sequences
##############################################################################
# We want to predict the next 10 time steps (each has 2 features).
# That means our target y will be shape (10 * 2) = 20 for each sample.

SEQ_LENGTH = 10          # how many past time steps we feed into LSTM
FORECAST_HORIZON = 10    # how many future steps to predict

def create_sequences_multistep(dataset, seq_length=10, forecast_horizon=10):
    X, y = [], []
    # We stop at len(dataset) - seq_length - forecast_horizon + 1
    for i in range(len(dataset) - seq_length - forecast_horizon + 1):
        # Past window
        past_window = dataset[i : i + seq_length]  # shape (seq_length, 2)
        
        # Next horizon steps (the future we want to predict)
        future = dataset[i + seq_length : i + seq_length + forecast_horizon]  
        # shape (forecast_horizon, 2) => e.g. (10, 2)
        
        # Flatten the future to match output_size=20
        future_flat = future.flatten()  # shape (forecast_horizon*2,)
        
        X.append(past_window)
        y.append(future_flat)
    return np.array(X), np.array(y)

X, Y = create_sequences_multistep(
    data_scaled, 
    seq_length=SEQ_LENGTH, 
    forecast_horizon=FORECAST_HORIZON
)

# For example, X.shape = (samples, 10, 2), Y.shape = (samples, 20)

##############################################################################
# 5. Train/Val/Test Split
##############################################################################
train_size = int(0.7 * len(X))
val_size   = int(0.1 * len(X))
test_size  = len(X) - train_size - val_size

X_train = X[:train_size]
y_train = Y[:train_size]

X_val   = X[train_size : train_size + val_size]
y_val   = Y[train_size : train_size + val_size]

X_test  = X[train_size + val_size:]
y_test  = Y[train_size + val_size:]

# Convert to tensors
X_train_tensors = torch.tensor(X_train, dtype=torch.float32)  # (train_size, 10, 2)
y_train_tensors = torch.tensor(y_train, dtype=torch.float32)  # (train_size, 20)

X_val_tensors   = torch.tensor(X_val,   dtype=torch.float32)
y_val_tensors   = torch.tensor(y_val,   dtype=torch.float32)

X_test_tensors  = torch.tensor(X_test,  dtype=torch.float32)
y_test_tensors  = torch.tensor(y_test,  dtype=torch.float32)

##############################################################################
# 6. Define the 3-Layer LSTM Model (Multi-step Output)
##############################################################################
class LSTMModel(nn.Module):
    def __init__(self, 
                 input_size,       # 2
                 hidden_size=50,   # 50 units in each LSTM layer
                 num_layers=3,     
                 output_size=20,   # 10 time steps * 2 features
                 dropout=0.2):
        super(LSTMModel, self).__init__()
        
        # First LSTM layer (returns the full sequence so the next LSTM can see all timesteps)
        self.lstm1 = nn.LSTM(input_size, hidden_size, batch_first=True)
        self.dropout1 = nn.Dropout(dropout)

        # Second LSTM layer (again returns the full sequence)
        self.lstm2 = nn.LSTM(hidden_size, hidden_size, batch_first=True)
        self.dropout2 = nn.Dropout(dropout)

        # Third LSTM layer (returns full sequence, we slice the last timestep)
        self.lstm3 = nn.LSTM(hidden_size, hidden_size, batch_first=True)
        self.dropout3 = nn.Dropout(dropout)

        # Final dense layer to map the last hidden state to your output size
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        # x shape: (batch, seq_length=10, input_size=2)
        
        # First layer
        out, _ = self.lstm1(x)
        out = self.dropout1(out)

        # Second layer
        out, _ = self.lstm2(out)
        out = self.dropout2(out)

        # Third layer
        out, _ = self.lstm3(out)
        out = self.dropout3(out)

        # We only need the last timestep’s output for final prediction
        out = out[:, -1, :]  # shape (batch, hidden_size)

        # Fully connected layer => shape (batch, output_size=20)
        out = self.fc(out)
        return out

##############################################################################
# 7. Instantiate Model
##############################################################################
model = LSTMModel(
    input_size=input_size,      # 2
    hidden_size=hidden_size,    # 64
    num_layers=num_layers,      # 3
    output_size=output_size,    # 20
    dropout=dropout             # 0.5
)

model.to(torch.device("cpu"))  # or "cuda" if you have a GPU

##############################################################################
# 8. Prepare for Training
##############################################################################
criterion = nn.MSELoss()  # Common for regression
optimizer = optim.Adam(model.parameters(), lr=learning_rate)  # 0.005

train_dataset = TensorDataset(X_train_tensors, y_train_tensors)
train_loader  = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)

##############################################################################
# 9. Training Loop
##############################################################################
for epoch in range(num_epochs):
    model.train()
    total_loss = 0
    
    for X_batch, y_batch in train_loader:
        optimizer.zero_grad()
        outputs = model(X_batch)  # shape (batch_size, 20)
        loss = criterion(outputs, y_batch)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
    
    avg_loss = total_loss / len(train_loader)
    
    # Optional: Validation
    model.eval()
    with torch.no_grad():
        val_outputs = model(X_val_tensors)
        val_loss = criterion(val_outputs, y_val_tensors).item()
    
    print(f"Epoch [{epoch+1}/{num_epochs}], Train Loss: {avg_loss:.6f}, Val Loss: {val_loss:.6f}")

##############################################################################
# 10. Evaluate on Test Set
##############################################################################
model.eval()
with torch.no_grad():
    test_preds = model(X_test_tensors)   # shape (test_size, 20)
    test_loss  = criterion(test_preds, y_test_tensors).item()

print(f"\nTest MSE Loss: {test_loss:.6f}")

# Convert predictions back to shape (test_size, FORECAST_HORIZON=10, 2)
test_preds_np = test_preds.numpy().reshape(-1, FORECAST_HORIZON, 2)
y_test_np     = y_test_tensors.numpy().reshape(-1, FORECAST_HORIZON, 2)

# (Optional) Inverse transform each 2D time step.
# Because we flattened for training, we need to invert the scaling carefully.
# If you used a single scaler for both columns, we can do:
test_preds_2d = test_preds_np.reshape(-1, 2)  # shape (test_size * 10, 2)
y_test_2d     = y_test_np.reshape(-1, 2)

test_preds_inv = scaler.inverse_transform(test_preds_2d)  # shape (total_predictions, 2)
y_test_inv     = scaler.inverse_transform(y_test_2d)

# Reshape back to (test_size, 10, 2)
test_preds_inv = test_preds_inv.reshape(-1, FORECAST_HORIZON, 2)
y_test_inv     = y_test_inv.reshape(-1, FORECAST_HORIZON, 2)

# Example: Plot the first test sample's 10 predicted points vs actual
plt.figure(figsize=(10, 4))
plt.plot(range(1, FORECAST_HORIZON+1), y_test_inv[0, :, 0], label="Actual col1")
plt.plot(range(1, FORECAST_HORIZON+1), test_preds_inv[0, :, 0], label="Pred col1")
plt.title("First Test Sample, col1 (10-step forecast)")
plt.legend()
plt.show()