Addressing imbalanced data

I’m developing a strategy to classify research papers as publishable or non-publishable, given a 2:1 imbalance ratio and 15 labeled papers. This involves examining text cleaning, representation, and various model approaches like generative, pipeline, and embedding models.

Handling imbalanced data

I’m thinking through how to handle data imbalance and section segmentation, possibly using bag-of-words. Discussing multi-class pipeline approaches, data augmentation, self-supervised methods, and classification strategies.

Thought about research paper classification strategies for 18 seconds

Below is a systematic breakdown of possible approaches and considerations for building a highly accurate and robust framework to classify research papers as “Publishable” or “Non-Publishable,” given the constraints and context you described:

1. Handling Limited and Imbalanced Labeled Data

1.1 Data Augmentation Strategies

1. Synthetic Document Generation / Data Augmentation

   • Paraphrasing: One could generate variants of the 5 “Non-Publishable” papers to increase the minority class. Although it’s tricky to ensure that paraphrased versions remain logically consistent with the original “Non-Publishable” label, it can help expand coverage of phrases and styles.
   • Section Swaps or Partial Text Replacement: For instance, you might keep the introduction and methodology the same but replace the conclusion with a rephrased or newly generated variant, thereby creating multiple versions of the same paper. This approach is more controlled than straightforward paraphrasing because you know exactly which sections you are modifying.

2. Document Splitting + Label Inheritance

   • You might split each labeled paper into coherent sections or even paragraphs. Each segment “inherits” the global label. Then you could train on this expanded set of labeled text segments. While you lose some context, you gain more training samples.
   • Careful: the label is at the “paper” level, not necessarily at the chunk level. Some smaller chunks might not by themselves indicate whether a paper is publishable. But as a data augmentation strategy, it can help models learn textual cues from smaller segments.

3. Active Learning or Semi-Supervised Learning

   • With 15 labeled papers, you might use an active learning loop:
     • Train an initial classifier on the small labeled set.
     • Use it to make predictions on the unlabeled 135 papers.
     • Select examples where the model is most uncertain (or most likely to be mislabeled) for human annotation.
   • This could help you quickly surface the most informative examples to be labeled, improving model performance with minimal labeling effort.

1.2 Handling Class Imbalance in Training

1. Oversampling / Undersampling

   • SMOTE for Text: Synthetic Minority Over-Sampling Technique (SMOTE) can be applied in the feature space (embedding vectors). If you create vector embeddings for each document (e.g., with BERT or Sentence Transformers), you can apply SMOTE on these embeddings to generate synthetic minority samples for the “Non-Publishable” class.
   • Random Oversampling or Undersampling: Simple approaches to either oversample the “Non-Publishable” class or undersample “Publishable.” Given the extremely small number of “Non-Publishable” papers, oversampling (or advanced synthetic generation) may be more suitable than undersampling (which would remove potentially valuable “Publishable” data).

2. Class Weighting / Focal Loss

   • Adjust the loss function by introducing higher weights for the minority class. In many deep learning frameworks, you can pass a parameter that sets the loss weighting for classes.
   • Alternatively, Focal Loss is useful in cases where the model might easily classify the majority class but struggles with the minority class.

2. Dealing with Large Documents of Varying Structure

2.1 Segmenting by Sections or Sub-Sections

• Splitting the documents by major sections (Abstract, Introduction, Methodology, etc.) can help isolate where certain “red flags” for non-publishability might appear. For instance:

  • Methodology section might reveal flawed or inappropriate methods without justification.
  • Results / Experiments section might show unrealistic claims or unsubstantiated results.
  • Introduction / Related Work might show a lack of clarity or organization.

• Issue: Not all papers have a uniform section structure.

  • Heuristic-based Section Detection: Use a simple set of regex patterns or markers (e.g., “1. Introduction”, “2. Background”, “Conclusion”, etc.) to segment text. Even if some papers don’t follow a standard template, you can still capture partial boundaries.
  • Dynamic Sub-Section Chunking: If some papers do not have clear headings or the headings vary significantly, you can chunk the paper into equal size segments (e.g., 1,000 tokens each). Then you can label or store the approximate position (beginning, middle, end), which correlates somewhat with Introduction, Methodology, Conclusion, etc.

2.2 Representation of Sections

• Section-level Embeddings: You could represent each section (or chunk) as an embedding using a large language model (e.g., BERT, RoBERTa, Sentence Transformers). This yields multiple embeddings per paper. You then aggregate these embeddings (e.g., by averaging them) into a single representation of the paper.
• Hierarchical Model: A hierarchical approach first encodes each section and then combines them with an attention mechanism. This can help the model learn that certain sections might hold more weight in publishability decisions.

2.3 Checking for Coherence and Methodological Soundness

• Some signals that a paper is “Non-Publishable”:
  1. Very short or incomplete methodology for a problem that clearly needs more detail.
  2. Results that are suspiciously high without rigorous justification.
  3. Inconsistencies or abrupt shifts in writing style that indicate disorganization.
• A rule-based or heuristic-based system (in parallel with an ML model) could flag suspicious patterns in these sections. Combined signals from these heuristics can augment your final classifier’s confidence.

3. Potential NLP Approaches

3.1 Pre-trained Transformers (Fine-Tuned)

Given the text-length and complexity, large pre-trained language models (PLMs) such as BERT, RoBERTa, DeBERTa, or Longformer (for longer context) can be used:

1. Longformer / BigBird

   • These architectures handle longer sequences than base BERT, which typically maxes out around 512 tokens. For entire research papers, you can feed more context without aggressively truncating.
   • You can do a [CLS] token classification or pool the final embeddings for classification.

2. Chunk-Level Approach with BERT

   • If you cannot use a long-context model, split the document into chunks of 512 tokens (or ~1,000 words, etc.) and pass each chunk through a BERT-based model.
   • Aggregate chunk-level embeddings or predictions (e.g., using mean or attention-based pooling) to form a final label for the entire paper.

3. Instruction-Finetuned or GPT-based Models Locally

   • While you cannot simply upload the entire dataset to GPT-4, you could leverage open-source instruction-finetuned models (like Llama 2, or Dolly, or other smaller GPT-like models) locally.
   • Fine-tune them on your 15 labeled papers (plus augmented data). This might be more feasible for capturing the notion of “publishability” because these models are fairly strong in domain reasoning if you shape your prompts or fine-tuning properly.

3.2 Classical ML + Feature Engineering

For smaller data scenarios, a simpler pipeline might be surprisingly effective:

1. Feature Extraction

   • TF-IDF: Standard textual features. Possibly weighted more heavily in the methodology and results sections.
   • Readability Scores: (e.g., Flesch-Kincaid) to capture clarity.
   • Semantic Similarity to Known “Non-Publishable” examples: Create an embedding for the entire text and measure the distance to embeddings from known non-publishable documents.
   • Keyword / Keyphrase extraction: Searching for “unsubstantiated,” “unproven,” or “lack of” in context might be a signal.

2. Classification Algorithms

   • Logistic Regression or Random Forest with class weighting.
   • Gradient Boosted Trees (XGBoost / LightGBM).
   • Even though these classical methods are overshadowed by large transformers on large data, they can perform competitively if your data is small and features are carefully engineered.

3.3 Multi-Task or Transfer Learning from “Topic Classification”

• You mentioned that the “Publishable” papers would fit into CVPR, NeurIPS, KDD, EMNLP, TMLR topics. One approach is:
  • Train a classifier to identify if a paper is about computer vision, NLP, data mining, etc. (You can do this with unlabeled data, because you can approximate the conference domain of the unlabeled paper by some heuristic or by an LLM prompt for domain classification. This step is somewhat easier to verify).
  • Then you use that knowledge as an auxiliary task, so the model “knows” what a typical CVPR or NeurIPS paper looks like. This can help it detect anomalies or major flaws in an “alleged” domain paper.

4. Practical Workflow Example

Below is an illustration of a pipeline that incorporates many of the above ideas:

1. Preprocessing & Document Splitting

   • Attempt to segment papers into major sections if possible (using heading detection or simple chunk-based segmentation otherwise).

2. Embeddings

   • Use a sentence-transformer or a Longformer variant to produce embeddings for each section (or chunk).

3. Aggregation

   • Combine embeddings for each chunk (using mean/attention).
   • Optionally, append a few engineered features (readability scores, average citations in references, etc.) to the final embedding vector.

4. Address Imbalance

   • Perform some oversampling (e.g., random oversampling for the 5 “Non-Publishable” papers) or synthetic minority data generation in the embedding space (SMOTE).
   • Additionally, use class weighting or Focal Loss during training.

5. Train Classifier

   • Fine-tune a robust model (e.g., a small MLP on top of embeddings, or fine-tune a transformer with a classification head).
   • Validate on a small hold-out from the 15 labeled samples or via cross-validation if you can’t spare a hold-out set.

6. Active Learning for Additional Labels

   • Run inference on the unlabeled 135 papers.
   • Identify 10–15 borderline examples that the model is least confident about.
   • Manually label them (if hackathon rules / time permits).
   • Retrain the model to further improve performance.

5. Potential Use of Large Language Models (LLMs) as Tools (Not Direct Classification)

While you cannot upload all data to GPT-4 or other LLMs directly for classification, you can still use them to aid certain subtasks:

1. Section Extraction / Title Identification

   • You can prompt an LLM to generate regular expressions or a procedure for detecting sections from a typical research paper. You implement that code locally and run it on your documents.
   • Or you can “ask” an LLM to propose heuristics for identifying a flawed methodology section. You can embed those heuristics into your code.

2. Topic Tagging

   • You can feed small abstracts or summarizations to an LLM to help determine if the paper belongs to CV, NLP, data mining, etc. This is short text, so it’s feasible to do partial queries. Then feed that domain label as an additional feature to your classifier.

3. Keyword Generation

   • Use an LLM to generate domain-specific “red flag” phrases for non-publishable content. Then you can build a dictionary-based detection system or incorporate it as a feature in your main model.

6. Ensuring Scalability and Consistency

• Scalability: By establishing a robust pipeline (section extraction → embeddings → classifier), you can easily add more labeled examples over time. If, in the future, you get thousands more labeled papers, you can seamlessly retrain the same pipeline at scale.
• Consistency Across Domains:
  • Because you’re dealing with multiple potential domains (CV, NLP, data mining, etc.), your model needs generalizable “red flags.”
  • Transfer learning from a generic language model (Longformer, BERT, etc.) already helps in capturing domain-invariant signals.
  • The additional domain classification step can also increase consistency by letting the model see typical domain structures.

7. Summary of Key Recommendations

1. Work with chunks or sections of the paper, using a hierarchical or chunk-based modeling approach (Longformer is ideal for large contexts).
2. Imbalance is a major issue; tackle it with oversampling (SMOTE in embedding space), class weighting, or focal loss.
3. Data augmentation: Explore paraphrasing or partial text modifications to increase the “Non-Publishable” set.
4. Active / Semi-Supervised Learning: Leverage the unlabeled 135 papers to refine your model iteratively.
5. Feature engineering: Combine advanced embeddings with simpler signals (keyword checks, readability, suspicious results).
6. LLMs can still be used as supportive tools (e.g., for section extraction heuristics, domain classification, or generating “red flag” keywords) without directly uploading the full dataset for classification.
7. Keep the pipeline modular so that you can easily experiment with different embedding models or classification heads.

Following these steps in a well-structured pipeline can yield a high-performing system, even with very limited labeled data and heavily imbalanced classes. By integrating data-centric approaches (like augmentation and active learning) with advanced NLP architectures (like Longformer/BERT for large text), you stand a strong chance of building a solution that meets the accuracy and F1 requirements for the hackathon.

Taking a closer look

I’m gathering info on unsupervised classification for 150 papers, considering heuristic-based section detection, hierarchical models, rule-based systems, semantic similarity, fine-tuning, and LLM-generated procedures.

Crafting the plan

Thinking through a Python-based approach with Kaggle libraries, using GPU/TPU. It involves PDF extraction, heuristic detection, unsupervised classification, hierarchical models, and rule-based or LLM-generated pattern generation.

Mapping the process

I’m planning to provide end-to-end Python code for each approach, focusing on unsupervised methods like clustering to spot suspicious patterns. Code blocks will be concise and illustrative.

Assessing documents

I’m pulling together code to chunk and store PDFs, apply unsupervised techniques like clustering with Sentence Transformers and KMeans for outlier detection, and develop a rule-based approach for spotting missing elements.

Assessing similarity

I’m crafting a semantic similarity snippet to match document embeddings, suggesting non-publishable examples if distances fall within a set threshold, and noting the need for GPU specification in Kaggle.

Thought about alternative approaches for classification for 9 seconds

Below is a detailed, end-to-end outline of how you might implement several of the ideas you highlighted—ranging from heuristic-based section detection, a hierarchical model design, rule-based “red flag” detection, unsupervised clustering, and leveraging semantic similarity. The goal is to provide a variety of methods you can try in your Kaggle environment. Where relevant, I’ve included code snippets (in Python) that you can adapt. I’ll also point out potential pitfalls or ways to improve your approach.

Table of Contents

1. Preprocessing & Section Extraction
   1.1 Basic Preprocessing vs. Context-Aware Preprocessing
   1.2 Heuristic-Based Section Detection

2. Unsupervised/Clustering Approach
   2.1 Embedding Generation (Sentence Transformers)
   2.2 K-Means / Hierarchical Clustering
   2.3 Interpreting Clusters & Using Minimal Labels

3. Hierarchical Model for Long Documents
   3.1 Chunking into Sections (or Subsections)
   3.2 Encoding & Combining Section Embeddings
   3.3 Fine-Tuning a Small Classifier Head (Optional)

4. Rule-Based “Red Flag” Detection
   4.1 Designing Heuristics
   4.2 Scoring & Combining with Other Methods

5. Semantic Similarity to Known Non-Publishable Papers
   5.1 Building a Similarity Index
   5.2 Threshold-Based Classification

6. Putting It All Together: Example Pipeline

7. Improving & Checking for Pitfalls

8. Complete Code Examples

   • 8.1 Section Extraction & Preprocessing
   • 8.2 Unsupervised Clustering (KMeans)
   • 8.3 Rule-Based Signals
   • 8.4 Semantic Similarity Scoring

Throughout, I’ll highlight ways to utilize your small set of 15 labeled papers effectively—even if it’s only at the end as a final “label assignment” step.

1. Preprocessing & Section Extraction

1.1 Basic Preprocessing vs. Context-Aware Preprocessing

What you’re currently doing:

def preprocess_text(text):
    text = re.sub(r'[^a-zA-Z0-9\s]', '', text)
    text = text.lower()
    tokens = word_tokenize(text)
    stop_words = set(stopwords.words('english'))
    tokens = [word for word in tokens if word not in stop_words]
    return ' '.join(tokens)

• Pros: Removes punctuation, lowercases, eliminates stopwords. This is a decent baseline for classical NLP.
• Cons:
  • You lose some valuable tokens like parentheses, percentages (useful for results sections: e.g., “accuracy = 99.9%!”).
  • You might break references to figures/tables (e.g., “Fig. 2.1” becomes “fig 21”).
  • Removing certain special tokens could hamper the detection of domain-specific notation.

Suggestions:

1. Keep punctuation in some form or at least keep certain symbols like %, =, maybe “( )” if relevant.
2. Consider lemmatization rather than just removing stopwords—sometimes lemmatization or a domain-specific tokenizer (like SciSpaCy for scientific text) can help.
3. Retain numeric tokens with some minimal cleaning. For example, if a paper claims “Accuracy of 150%,” that’s obviously suspicious and might be relevant to detect.

1.2 Heuristic-Based Section Detection

Many research papers follow a rough structure with headings like:

• Abstract
• Introduction
• Related Work / Background
• Methodology / Experiments
• Results
• Conclusion / Future Work

Heuristic approach:

• Use regex patterns to capture lines that look like headings (e.g., “^([0-9]+.)+\s+[A-Z].+$” or just “^(abstract|introduction|conclusion|references)[^a-zA-Z]+” ignoring case).
• If you can’t find a heading, split by large chunks (e.g., 1,000–2,000 words) as a fallback.

Below is a conceptual snippet for Kaggle (Python) to detect sections:

import re

def split_into_sections(pdf_text):
    # Possible common headings to look for
    headings = [
        r"(abstract)\b",
        r"(introduction)\b",
        r"(background|related\s?work)\b",
        r"(methodology|methods|experimental\s?setup)\b",
        r"(results|experiments)\b",
        r"(discussion)\b",
        r"(conclusion|conclusions)\b",
        r"(references)\b"
    ]

    # We'll store tuples of (section_name, section_text)
    sections = []
    current_section_name = "Unlabeled"
    current_section_text = []

    lines = pdf_text.split('\n')
    for line in lines:
        line_lower = line.lower().strip()
        matched_heading = None
        
        for hd in headings:
            if re.match(hd, line_lower):
                matched_heading = line.strip()
                break

        if matched_heading:
            # Save previous section if we have any text
            if current_section_text:
                sections.append((current_section_name, "\n".join(current_section_text)))
            # Start a new section
            current_section_name = matched_heading
            current_section_text = []
        else:
            current_section_text.append(line)

    # Add the last collected section if any
    if current_section_text:
        sections.append((current_section_name, "\n".join(current_section_text)))

    return sections

# Example usage
# pdf_text = "Your extracted text from pdfplumber"
# sections = split_into_sections(pdf_text)
# for sec_name, sec_content in sections:
#     print(sec_name, len(sec_content.split()))

What to do next:

• You can pass each sec_content through your preprocess_text or a more domain-aware pipeline.
• This helps your unsupervised approaches or hierarchical model treat each major section somewhat separately.

2. Unsupervised/Clustering Approach

2.1 Embedding Generation (Sentence Transformers)

1. Sentence Transformers (e.g., all-mpnet-base-v2, all-distilroberta-v1, or a domain-specific model like allenai/scibert_scivocab_uncased) can create embeddings for text segments.
2. For each of your 150 documents, produce one embedding for each major section (or chunk). Then either average them to get a single “document embedding,” or keep them separate for hierarchical modeling.

2.2 K-Means / Hierarchical Clustering

• K-Means is a straightforward choice if you want a fixed number of clusters. However, deciding the number of clusters is tricky (especially if you have 2 classes—“Publishable” vs. “Non-Publishable”—but the actual text may form more than 2 clusters).
• Hierarchical clustering or HDBSCAN can discover an arbitrary number of clusters and potentially identify outliers.

2.3 Interpreting Clusters & Using Minimal Labels

1. Cluster the 150 documents (using their embeddings) into, say, $k = 5$ or $k = 6$ clusters.
2. Look at which cluster your 5 known “Non-Publishable” papers end up in. If they cluster together or in smaller distinct subgroups, that is a strong signal.
3. Optionally, label entire clusters as “likely publishable” or “likely non-publishable” if they strongly align with your known labeled docs. If a cluster has no known non-publishable examples, treat it carefully or look for rule-based signals to see if it might be “suspect.”

3. Hierarchical Model for Long Documents

3.1 Chunking into Sections

By splitting papers into sections or ~1,000-word chunks:

def chunk_text(text, chunk_size=1000):
    words = text.split()
    for i in range(0, len(words), chunk_size):
        yield ' '.join(words[i:i+chunk_size])

Then process each chunk with a transformer (like SciBERT) to obtain a vector.

3.2 Encoding & Combining Section Embeddings

One approach (a “hierarchical” approach) is:

1. Encode each chunk with a transformer → you get chunk_embeddings of shape [num_chunks, embedding_dim].
2. Combine chunk_embeddings with an attention or simple average. For instance: $$\text{paper\_embedding} = \text{mean}(chunk\_embeddings)$$
3. Use the final paper_embedding for either clustering (unsupervised) or as input to a small neural classifier if you do want to use your labeled data.

3.3 Fine-Tuning a Small Classifier Head (Optional)

Even if you have only 15 labeled documents, you could:

1. Generate chunk-level embeddings for all 150 documents.
2. Use the 15 labeled docs to train a tiny classifier (like an MLP or logistic regression) that takes the aggregated embedding as input.
3. Apply that classifier to all 150 docs.
4. Because your labeled data is tiny, you’ll risk overfitting—so you might rely on cross-validation or the cluster-based signals to calibrate or confirm.

4. Rule-Based “Red Flag” Detection

4.1 Designing Heuristics

Some potential signals for “Non-Publishable”:

1. Suspicious High-Level Claims: Searching for tokens like “100% accuracy,” “state-of-the-art with no baseline,” etc.
2. Weak or Missing Methodology: If the “Methodology” section has fewer than X words, or if certain keywords (“experiment,” “implementation,” “dataset,” “train,” “test”) are missing.
3. Lack of references or citations: If the “References” or “Bibliography” section is absent or very short.

4.2 Scoring & Combining with Other Methods

You could create a simple “red flag” score for each paper:

• For each rule triggered, add +1 (or +some weighting).
• At the end, if total “red flag” score exceeds a threshold, you flag the paper as suspicious.

This approach can run in parallel with your unsupervised model. You might ultimately label a paper “non-publishable” if:

• It’s in the same cluster as known non-publishable docs, and
• It exhibits at least 1–2 rule-based red flags.

5. Semantic Similarity to Known Non-Publishable Papers

5.1 Building a Similarity Index

1. Compute the average embedding of your 5 known non-publishable papers, call it nonpub_centroid.
2. For each unlabeled paper, compute its embedding and compute cosine similarity to nonpub_centroid.
3. If the similarity is above a certain threshold, the paper is considered suspicious. Alternatively, keep the top N “closest” papers as candidates for “non-publishable.”

5.2 Threshold-Based Classification

A simple approach:

• You might pick a similarity threshold by checking your known 5 non-publishable vs. 10 publishable papers.
• If your 5 known non-publishable have an average similarity of 0.8 to the centroid, while the 10 publishable ones have 0.5, then you can choose ~0.65 as a cutoff.

6. Putting It All Together: Example Pipeline

1. Extract PDF text using pdfplumber.
2. Heuristic-based section detection (if possible).
3. Preprocess each section (decide on minimal vs. domain-aware cleaning).
4. Generate embeddings (e.g., using a SentenceTransformer model).
5. Combine section embeddings → one doc embedding (mean or attention).
6. Unsupervised clustering or semantic similarity approach.
7. Rule-based checks for suspicious patterns.
8. (Optional) Small classifier using your 15 labeled docs.

7. Improving & Checking for Pitfalls

• Overfitting to the 15 labeled docs:

  • If you do any supervised training, consider cross-validation or a simple “leave-one-out” approach to get a rough measure.
  • Or rely primarily on unsupervised methods, then do a final check with the 15 labeled docs to see if it aligns with your intuition.

• Inconsistent Section Headings:

  • Many papers might not follow standard headings. You might need fallback chunking.

• Large Variation in Paper Length:

  • Some papers might be 1–2 pages short workshop papers, others 20+ pages. Make sure your chunking logic can handle extremes.

• GPU/TPU Usage:

  • In Kaggle, you can enable a GPU by going to Settings > Accelerator. Using a transformer model for embeddings typically works well on GPUs.

8. Complete Code Examples

Below are some consolidated code snippets (in Python) that you can adapt in a Kaggle notebook. I’ll assume you’ve already installed pdfplumber and sentence-transformers. (If not, you can do !pip install pdfplumber sentence-transformers at the top of your Kaggle notebook.)

8.1 Section Extraction & Preprocessing

import re
import pdfplumber
import torch
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

# Make sure to download NLTK data if not already present
# nltk.download('punkt')
# nltk.download('stopwords')

def extract_text_from_pdf(pdf_path):
    """Extract all text from a PDF using pdfplumber."""
    pdf_text = ""
    with pdfplumber.open(pdf_path) as pdf:
        for page in pdf.pages:
            page_text = page.extract_text()
            if page_text:
                pdf_text += page_text + "\n"
    return pdf_text

def custom_preprocess(text):
    """
    A slightly more cautious preprocessing that:
    - Keeps numbers, percentage signs,
    - Lowercases,
    - Removes some punctuation but keeps essential symbols.
    """
    # Lowercase
    text = text.lower()

    # Keep only certain punctuation: letters, numbers, whitespace, punctuation like '%', '=', etc.
    text = re.sub(r'[^a-z0-9%\=$$\.\,\s]', '', text)

    # Tokenize
    tokens = word_tokenize(text)

    # Remove stopwords (optional)
    stop_words = set(stopwords.words('english'))
    tokens = [t for t in tokens if t not in stop_words]

    return ' '.join(tokens)

def split_into_sections(pdf_text):
    headings = [
        r"(abstract)\b",
        r"(introduction)\b",
        r"(background|related\s?work)\b",
        r"(methodology|methods|experimental\s?setup)\b",
        r"(results|experiments)\b",
        r"(discussion)\b",
        r"(conclusion|conclusions)\b",
        r"(references)\b"
    ]

    sections = []
    current_section_name = "Unlabeled"
    current_section_text = []

    lines = pdf_text.split('\n')
    for line in lines:
        line_lower = line.strip().lower()
        matched_heading = None
        for hd in headings:
            if re.match(hd, line_lower):
                matched_heading = line.strip()
                break

        if matched_heading:
            if current_section_text:
                sections.append((current_section_name, "\n".join(current_section_text)))
            current_section_name = matched_heading
            current_section_text = []
        else:
            current_section_text.append(line)

    if current_section_text:
        sections.append((current_section_name, "\n".join(current_section_text)))

    return sections

Usage in a Kaggle environment:

# Suppose you have a folder "papers" with PDF files
import os

pdf_dir = "./papers"
documents = []

for filename in os.listdir(pdf_dir):
    if filename.endswith(".pdf"):
        path = os.path.join(pdf_dir, filename)
        raw_text = extract_text_from_pdf(path)
        sections = split_into_sections(raw_text)

        # Preprocess each section
        processed_sections = []
        for sec_name, sec_text in sections:
            processed_text = custom_preprocess(sec_text)
            processed_sections.append((sec_name, processed_text))

        # Store the document data
        documents.append({
            'filename': filename,
            'sections': processed_sections
        })

8.2 Unsupervised Clustering (KMeans)

Below is a simple example of clustering the average embedding of each paper. We’ll use Sentence Transformers to get embeddings.

!pip install sentence-transformers

from sentence_transformers import SentenceTransformer
from sklearn.cluster import KMeans
import numpy as np

# Load a pre-trained or domain-specific model (SciBERT, etc.)
model_name = 'all-mpnet-base-v2'  # or 'allenai/scibert_scivocab_uncased'
device = 'cuda' if torch.cuda.is_available() else 'cpu'
embed_model = SentenceTransformer(model_name, device=device)

def get_document_embedding(processed_sections):
    """
    Generate embeddings for each section, then average them.
    processed_sections: list of (sec_name, text) after preprocessing.
    """
    section_texts = [sec_text for _, sec_text in processed_sections if sec_text.strip()]
    if not section_texts:
        # fallback if there's no text
        return np.zeros(768)

    # Encode the entire batch
    embeddings = embed_model.encode(section_texts, convert_to_numpy=True)
    doc_embedding = np.mean(embeddings, axis=0)
    return doc_embedding

# Compute embeddings for all documents
doc_embeddings = []
for doc in documents:
    embedding = get_document_embedding(doc['sections'])
    doc_embeddings.append(embedding)
doc_embeddings = np.array(doc_embeddings)

# K-Means Clustering
num_clusters = 5  # arbitrary choice; you can experiment
kmeans = KMeans(n_clusters=num_clusters, random_state=42)
labels = kmeans.fit_predict(doc_embeddings)

for i, doc in enumerate(documents):
    doc['cluster_label'] = labels[i]

# Inspect the cluster distribution
from collections import Counter
cluster_counts = Counter(labels)
print(cluster_counts)

Next Steps:

• Check which clusters your known 5 non-publishable (if you do know their filenames) fall into.
• Potentially label entire clusters as suspicious or safe, or use rule-based signals to refine.

8.3 Rule-Based Signals

Create a few simple functions to detect “red flags,” e.g.:

def count_red_flags(processed_sections):
    flags = 0

    # 1) Check for unrealistic claims: e.g., "100% accuracy"
    text_joined = " ".join(sec_text for _, sec_text in processed_sections)
    if '100% accuracy' in text_joined or '100 percent accuracy' in text_joined:
        flags += 1

    # 2) Check if Methodology section is too short
    for sec_name, sec_text in processed_sections:
        if 'method' in sec_name.lower():
            # Adjust threshold to your taste
            if len(sec_text.split()) < 100:
                flags += 1

    # 3) Check if there's a references section at all
    has_references = any('references' in sec_name.lower() for sec_name, _ in processed_sections)
    if not has_references:
        flags += 1

    return flags

# Example usage
for doc in documents:
    doc['red_flag_score'] = count_red_flags(doc['sections'])

# Then you could set a threshold
threshold = 1
for doc in documents:
    if doc['red_flag_score'] >= threshold:
        doc['suspicious'] = True
    else:
        doc['suspicious'] = False

8.4 Semantic Similarity Scoring

If you have 5 known non-publishable papers:

known_nonpub_embeddings = []
for doc in documents:
    if doc['filename'] in known_nonpub_filenames:  # e.g. the 5 known non-publishable
        emb = get_document_embedding(doc['sections'])
        known_nonpub_embeddings.append(emb)
nonpub_centroid = np.mean(known_nonpub_embeddings, axis=0)

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# Score each doc
for doc in documents:
    emb = get_document_embedding(doc['sections'])
    sim = cosine_similarity(emb, nonpub_centroid)
    doc['similarity_to_nonpub'] = sim

Then set a threshold:

# For instance, see the distribution of similarity_to_nonpub for known publishable vs. known non-publishable
# This helps you pick a threshold
threshold = 0.7

for doc in documents:
    doc['maybe_nonpub'] = (doc['similarity_to_nonpub'] > threshold)

Putting It All Together in a Step-by-Step Manner

Below is a condensed version of how this might look in an actual Kaggle Notebook with minimal commentary:

# Step 0: Install dependencies (if needed)
!pip install pdfplumber sentence-transformers

import os
import pdfplumber
import re
import torch
import nltk
import numpy as np
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from sentence_transformers import SentenceTransformer
from sklearn.cluster import KMeans
from collections import Counter

device = 'cuda' if torch.cuda.is_available() else 'cpu'
model_name = 'all-mpnet-base-v2'
embed_model = SentenceTransformer(model_name, device=device)

# Download NLTK data
nltk.download('punkt')
nltk.download('stopwords')

def extract_text_from_pdf(pdf_path):
    pdf_text = ""
    with pdfplumber.open(pdf_path) as pdf:
        for page in pdf.pages:
            page_text = page.extract_text()
            if page_text:
                pdf_text += page_text + "\n"
    return pdf_text

def custom_preprocess(text):
    text = text.lower()
    # Keep certain punctuation
    text = re.sub(r'[^a-z0-9%\=$$\.\,\s]', '', text)
    tokens = word_tokenize(text)
    stop_words = set(stopwords.words('english'))
    tokens = [t for t in tokens if t not in stop_words]
    return ' '.join(tokens)

def split_into_sections(pdf_text):
    headings = [
        r"(abstract)\b",
        r"(introduction)\b",
        r"(background|related\s?work)\b",
        r"(methodology|methods|experimental\s?setup)\b",
        r"(results|experiments)\b",
        r"(discussion)\b",
        r"(conclusion|conclusions)\b",
        r"(references)\b"
    ]

    sections = []
    current_section_name = "Unlabeled"
    current_section_text = []

    lines = pdf_text.split('\n')
    for line in lines:
        line_lower = line.strip().lower()
        matched_heading = None
        for hd in headings:
            if re.match(hd, line_lower):
                matched_heading = line.strip()
                break

        if matched_heading:
            if current_section_text:
                sections.append((current_section_name, "\n".join(current_section_text)))
            current_section_name = matched_heading
            current_section_text = []
        else:
            current_section_text.append(line)

    if current_section_text:
        sections.append((current_section_name, "\n".join(current_section_text)))

    return sections

def get_document_embedding(processed_sections):
    section_texts = [sec_text for _, sec_text in processed_sections if sec_text.strip()]
    if not section_texts:
        return np.zeros(768)
    embeddings = embed_model.encode(section_texts, convert_to_numpy=True)
    return np.mean(embeddings, axis=0)

def count_red_flags(processed_sections):
    flags = 0
    text_joined = " ".join(sec_text for _, sec_text in processed_sections)

    # Example rule: 100% accuracy
    if '100% accuracy' in text_joined or '100 percent accuracy' in text_joined:
        flags += 1

    # Example rule: short methodology section
    for sec_name, sec_text in processed_sections:
        if 'method' in sec_name.lower():
            if len(sec_text.split()) < 100:
                flags += 1

    # Example rule: no references
    has_references = any('references' in sec_name.lower() for sec_name, _ in processed_sections)
    if not has_references:
        flags += 1

    return flags


# ===== MAIN PIPELINE =====
pdf_dir = "./papers"
documents = []

for filename in os.listdir(pdf_dir):
    if filename.endswith(".pdf"):
        path = os.path.join(pdf_dir, filename)
        raw_text = extract_text_from_pdf(path)
        sections = split_into_sections(raw_text)

        # Preprocess each section
        processed_sections = []
        for sec_name, sec_text in sections:
            processed_text = custom_preprocess(sec_text)
            processed_sections.append((sec_name, processed_text))

        documents.append({
            'filename': filename,
            'sections': processed_sections
        })

# Compute embeddings for each doc
doc_embeddings = []
for doc in documents:
    emb = get_document_embedding(doc['sections'])
    doc_embeddings.append(emb)
doc_embeddings = np.array(doc_embeddings)

# KMeans clustering
num_clusters = 5
kmeans = KMeans(n_clusters=num_clusters, random_state=42)
labels = kmeans.fit_predict(doc_embeddings)

for i, doc in enumerate(documents):
    doc['cluster_label'] = int(labels[i])
    doc['red_flag_score'] = count_red_flags(doc['sections'])

cluster_counts = Counter(labels)
print("Cluster distribution:", cluster_counts)

# Suppose we know the filenames of the 5 known non-publishable
known_nonpub_filenames = ["nonpub1.pdf", "nonpub2.pdf", "nonpub3.pdf", "nonpub4.pdf", "nonpub5.pdf"]
known_nonpub_embeddings = []

for doc in documents:
    if doc['filename'] in known_nonpub_filenames:
        known_nonpub_embeddings.append(get_document_embedding(doc['sections']))

nonpub_centroid = np.mean(known_nonpub_embeddings, axis=0)

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

for doc in documents:
    emb = get_document_embedding(doc['sections'])
    sim = cosine_similarity(emb, nonpub_centroid)
    doc['similarity_to_nonpub'] = float(sim)

# Example threshold for suspicious
similarity_threshold = 0.7
red_flag_threshold = 1

for doc in documents:
    doc['maybe_nonpub'] = (doc['similarity_to_nonpub'] > similarity_threshold) \
                          or (doc['red_flag_score'] >= red_flag_threshold)

# Inspect final results
for doc in documents:
    print(doc['filename'], doc['cluster_label'], doc['red_flag_score'], doc['similarity_to_nonpub'], doc['maybe_nonpub'])

1. Load PDFs → Extract text
2. Heuristics to split into sections
3. Preprocessing that’s a bit more selective
4. Compute embeddings for each doc (averaging section embeddings)
5. Clustering (K-Means) → see if certain clusters align with known non-publishable examples
6. Rule-based flags → short methodology, missing references, suspicious claims
7. Semantic similarity to known non-publishable papers (centroid)
8. Combine signals to produce a final guess (maybe_nonpub)

Final Notes & Key Takeaways

1. Unsupervised + Heuristics: Since you have such limited labeled data, leaning on unsupervised clustering + rule-based detection is a solid approach.
2. Section Extraction: Imperfect headings are still better than none. For papers that don’t match headings, fallback chunking is a must.
3. Semantic Similarity: Quick way to see how close new papers are to known “bad” examples.
4. Small Classifier: If time permits, you can still attempt a small MLP on top of embeddings, but be wary of overfitting with only 15 labeled examples.
5. Spot-Check: Even with unsupervised methods, you still want to do some manual inspection of the clusters and flagged papers.
6. Continual Improvement: If you can incrementally label more papers (even if it’s partial or just verifying cluster assignments), you’ll see big gains in performance.

I hope this gives you a more comprehensive roadmap—and enough code to try out each approach in Kaggle. Feel free to tweak thresholds, the choice of embedding model, clustering method, and rule-based patterns to adapt to your specific corpus. Above all, remember that your domain knowledge about “what makes a paper unpublishable” can be translated into powerful heuristics that complement the unsupervised features.

Framing the approach

I’m setting up for a KMeans visualization by reducing dimensionality with PCA, then creating a scatter plot to color or label clusters, highlighting specific examples.

Visualizing similarities

I’m preparing to create a histogram or density curve to show cosine similarity distributions, using colors or shapes to highlight 15 specific documents and their unique characteristics.

Thought for 4 seconds

Below is a structured walk-through of how to visualize the clustering results, the cosine similarity distribution, and the red flag scores for your documents. I’ve also included code snippets that:

• Highlight the 15 labeled PDFs (R001.pdf to R015.pdf), with R001–R005 labeled as class 0 and R006–R015 labeled as class 1.
• Plot the clustering in 2D using either PCA or t-SNE.
• Draw a histogram (or KDE) of cosine similarity values, highlighting the labeled ones.
• Provide additional “red flag” heuristics and illustrate how to visualize their distribution.

Where possible, I’ve included commentary for improvements or alternative approaches.

1. Visualizing the K-Means Clusters

Since doc_embeddings is likely high-dimensional (e.g., 768 dimensions from a transformer), we’ll reduce the dimensionality to 2D for plotting. You can use PCA, t-SNE, or UMAP. Below, I’ll show PCA for simplicity, but feel free to replace it with t-SNE or UMAP.

import matplotlib.pyplot as plt
import numpy as np
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE

# We have: 
#   doc_embeddings (shape: [num_documents, embedding_dim])
#   documents (a list of dicts with 'filename' and 'cluster_label')

# Let's define our labeled sets
labeled_files_class0 = set([f"R00{i}.pdf" for i in range(1,6)])  # R001 to R005
labeled_files_class1 = set([f"R00{i}.pdf" for i in range(6,16)]) # R006 to R015

# (1) PCA for 2D visualization
pca = PCA(n_components=2, random_state=42)
reduced_pca = pca.fit_transform(doc_embeddings)

# (2) Alternatively, you can try t-SNE (comment out PCA if you want this)
# tsne = TSNE(n_components=2, random_state=42)
# reduced_tsne = tsne.fit_transform(doc_embeddings)
# Then use reduced_tsne instead of reduced_pca below.

# We'll use reduced_pca here for demonstration.
x_coords = reduced_pca[:, 0]
y_coords = reduced_pca[:, 1]

# Create a figure for the cluster visualization
plt.figure(figsize=(10, 7))

# Plot each document as a point, colored by cluster_label
clusters = [doc['cluster_label'] for doc in documents]
scatter = plt.scatter(x_coords, y_coords, c=clusters, cmap='tab10', alpha=0.6)

plt.title("K-Means Clusters (PCA 2D Projection)")
plt.xlabel("Principal Component 1")
plt.ylabel("Principal Component 2")

# Now let's highlight or label the 15 known PDFs
for i, doc in enumerate(documents):
    filename = doc['filename']
    # If this doc is one of the 15 labeled
    if filename in labeled_files_class0 or filename in labeled_files_class1:
        # We'll put a black edge or a special marker
        plt.scatter(x_coords[i], y_coords[i], 
                    c='none', edgecolors='black', 
                    marker='o', s=100, linewidths=1.5)

        # Optionally, annotate text
        # e.g., display 'C0' or 'C1' for class 0 or class 1
        if filename in labeled_files_class0:
            label_text = "C0"
        else:
            label_text = "C1"
        plt.text(x_coords[i]+0.02, y_coords[i]+0.02, label_text, fontsize=9)

# Add a legend for the cluster colors
# We'll add it by mapping each cluster label to a color patch
unique_clusters = list(sorted(set(clusters)))
handles = scatter.legend_elements()[0]
labels_for_legend = [f"Cluster {uc}" for uc in unique_clusters]
plt.legend(handles, labels_for_legend, title="Clusters", loc="best")

plt.show()

Explanation / Notes

• We used PCA to reduce to 2D, then plotted each document as a scatter point colored by its cluster_label.
• The 15 labeled PDFs (R001-R005 are class 0, R006-R015 are class 1) are drawn with a black edge around the marker, and we label them in text as “C0” or “C1.”
• This way, you can visually see how your labeled examples are distributed across clusters.

If you want a more advanced or non-linear reduction (like t-SNE or UMAP), just uncomment the relevant lines, and refer to reduced_tsne[:, 0] and reduced_tsne[:, 1] for x and y coordinates.

2. Visualizing Cosine Similarities

You have code that computes doc['similarity_to_nonpub'] for each document. Let’s plot a histogram (or KDE plot) of these similarity scores and highlight the 15 labeled PDFs.

import seaborn as sns

# Extract similarity scores for all docs
similarities = [doc['similarity_to_nonpub'] for doc in documents]

# Make a list of labeled similarities, plus the label for each
similarities_labeled_c0 = []
similarities_labeled_c1 = []
similarities_unlabeled = []

for i, doc in enumerate(documents):
    sim_val = doc['similarity_to_nonpub']
    filename = doc['filename']
    if filename in labeled_files_class0:
        similarities_labeled_c0.append(sim_val)
    elif filename in labeled_files_class1:
        similarities_labeled_c1.append(sim_val)
    else:
        similarities_unlabeled.append(sim_val)

plt.figure(figsize=(10, 6))

# Plot distribution with seaborn's histplot or kdeplot
sns.histplot(similarities, kde=True, color='gray', alpha=0.4, label='All Documents')

# Plot labeled class 0
sns.histplot(similarities_labeled_c0, color='red', alpha=0.8, kde=True, label='Class 0 (R001-R005)')

# Plot labeled class 1
sns.histplot(similarities_labeled_c1, color='blue', alpha=0.6, kde=True, label='Class 1 (R006-R015)')

plt.title("Cosine Similarity to Non-Publishable Centroid")
plt.xlabel("Similarity Score")
plt.ylabel("Count")

plt.legend(loc='best')
plt.show()

Explanation / Notes

• We separate out the distributions for:
  • All documents (gray)
  • Class 0 labeled docs (red)
  • Class 1 labeled docs (blue)
• This helps you see whether your “Non-Publishable” examples (class 0) indeed have higher similarity to the nonpub_centroid than the “Publishable” ones.

Additional Variation

• If you prefer a KDE-only approach (smooth density curve), you can use sns.kdeplot(...) instead of sns.histplot(kde=True, ...).
• If you want to overlay them differently, you can create separate subplots or adjust transparency.

3. More Red Flag Rules

You mentioned that having or not having a references section wasn’t always accurate. Below are some alternative heuristics:

1. Extremely Short Paper: If the overall word count is below some threshold (e.g., < 1000 words), it might be suspicious for a full conference paper.
2. No Mention of Baseline / Comparison: For example, check if words like “baseline,” “comparison,” or “benchmark” appear in the method/results sections. A lack of any mention might be suspicious.
3. No Mention of Datasets / Data: Many (but not all) publishable ML papers talk about dataset(s). If the text never mentions “dataset,” “corpus,” or “data used,” that might be suspicious.
4. No Mention of Metrics: E.g., no mention of “accuracy,” “precision,” “recall,” “auc,” “f1,” “mse,” “rmse,” etc. in the results section.
5. Excessive Self-Citations or “I” / “we” usage in the methodology (less formal). This is more subjective.
6. Inconsistent or Contradictory numeric results: If the results section claims two different accuracies for the same experiment. (This is more complex to implement and requires deeper NLP to parse the content.)

Below is an updated example of a “red flag” function using some of these additional heuristics:

def count_red_flags(processed_sections):
    flags = 0
    full_text = " ".join(sec_text for _, sec_text in processed_sections)
    word_count = len(full_text.split())

    # 1) Very short paper
    if word_count < 1000:
        flags += 1

    # 2) Check if baseline or comparison keywords are absent
    if ("baseline" not in full_text) and ("compare" not in full_text) and ("benchmark" not in full_text):
        flags += 1

    # 3) Check if there's any mention of data or dataset
    if ("data" not in full_text) and ("dataset" not in full_text) and ("corpus" not in full_text):
        flags += 1

    # 4) No mention of common evaluation metrics
    common_metrics = ["accuracy", "precision", "recall", "f1", "auc", "mse", "rmse"]
    if not any(metric in full_text for metric in common_metrics):
        flags += 1

    return flags

Things to Keep in Mind

• The above rules are heuristics and can yield false positives. For instance, a theoretical paper might not mention “dataset,” or a short paper might still be legitimate (like a short workshop paper).
• Always tailor these rules to your actual data. Adjust thresholds and keyword sets as you learn more about your corpus.

4. Visualizing Red Flag Scores

Similar to the cosine similarity distribution, you might want to plot a histogram of red_flag_score across all documents. We’ll again highlight the labeled ones.

import seaborn as sns

red_flag_scores = [doc['red_flag_score'] for doc in documents]
scores_labeled_c0 = []
scores_labeled_c1 = []
scores_unlabeled = []

for doc in documents:
    score_val = doc['red_flag_score']
    filename = doc['filename']
    if filename in labeled_files_class0:
        scores_labeled_c0.append(score_val)
    elif filename in labeled_files_class1:
        scores_labeled_c1.append(score_val)
    else:
        scores_unlabeled.append(score_val)

plt.figure(figsize=(10, 6))

sns.histplot(red_flag_scores, color='gray', alpha=0.4, binwidth=1, label='All Documents')
sns.histplot(scores_labeled_c0, color='red', alpha=0.8, binwidth=1, label='Class 0 (R001-R005)')
sns.histplot(scores_labeled_c1, color='blue', alpha=0.6, binwidth=1, label='Class 1 (R006-R015)')

plt.title("Red Flag Score Distribution")
plt.xlabel("Red Flag Score (count of triggered heuristics)")
plt.ylabel("Number of Papers")
plt.legend(loc='best')
plt.show()

This will give you a sense of how many rules each document is triggering, and whether your known class 0 or class 1 docs align with high or low red-flag counts.

5. Suggested Improvements

1. Refine & Validate Heuristics
   • Inspect actual “publishable” vs. “non-publishable” papers to confirm that your rules reflect reality. For example, if some well-written short paper triggers 3 red flags, maybe you need to adjust thresholds.
2. Combine Scores
   • You can combine red flag scores with clustering or similarity. For instance, mark a paper as “non-publishable” if it triggers >2 red flags and has a high similarity to a known non-publishable centroid.
3. Add Some Manual Spot-Checking
   • Because you have only 15 labeled papers, additional manual checks or partial labeling might help refine your thresholds.
4. Consider More Advanced Dimensionality Reduction
   • t-SNE or UMAP might cluster your embeddings more intuitively than PCA for non-linear separations.
5. Use a Small Classifier as a Final Step
   • Even though you have a small labeled set, you can do a simple logistic regression or MLP with strong regularization or cross-validation. You might combine features like “paper_embedding,” “red_flag_score,” and “similarity_to_nonpub.” Then use cross-validation on your 15 labeled docs to pick hyperparameters.
6. Look for Contradictory Results
   • Potentially parse numeric results to see if there are contradictions (like claiming “accuracy of 90%” in one paragraph and “accuracy of 95%” in another for the same experiment). This is advanced but can be a strong heuristic.

Summary

• Clustering Visualization: Use PCA/t-SNE on doc_embeddings, color by cluster, highlight labeled docs.
• Cosine Similarity Distribution: Plot a histogram or KDE, highlight labeled docs.
• Red Flag Heuristics: Expand your set of rules to cover more typical “publishability” signals (baselines, data usage, metrics, etc.).
• Red Flag Distribution: Similarly, plot a histogram of how many heuristics each paper triggers and see if your labeled examples match your intuition.
• Refinement: Iteratively refine thresholds and rules after examining outliers.

By combining these visualizations, you’ll get a much better understanding of how your dataset is structured, how your known “non-publishable” vs. “publishable” examples fit in, and which heuristics or similarity thresholds might be most effective.

The issue you're describing likely stems from how the labeled files (R001 to R015) are matched against the filenames in your documents list. If there’s any mismatch in how the filenames are stored (e.g., missing .pdf extensions, inconsistent capitalization, or whitespace issues), some of the labeled files might not be correctly identified, resulting in only a subset being visualized.

Here’s a step-by-step rectification and improved code to ensure all 15 labeled files are visualized properly:

Step 1: Verify File Matching Logic

Ensure that the filenames for your labeled files are correctly matched against the filenames in your documents list. Normalize both sets of filenames to avoid mismatches.

# Define labeled files with consistent formatting
labeled_files_class0 = {f"R00{i}.pdf" for i in range(1, 6)}  # R001-R005
labeled_files_class1 = {f"R00{i}.pdf" for i in range(6, 16)}  # R006-R015

# Normalize filenames in the documents list for comparison
for doc in documents:
    doc['normalized_filename'] = doc['filename'].strip().lower()  # Normalize filenames

# Ensure the labeled sets are also normalized
labeled_files_class0 = {f.strip().lower() for f in labeled_files_class0}
labeled_files_class1 = {f.strip().lower() for f in labeled_files_class1}

Step 2: Correct Clustering Visualization Code

Update the clustering visualization code to ensure all labeled files are highlighted properly:

# PCA for dimensionality reduction
pca = PCA(n_components=2, random_state=42)
reduced_pca = pca.fit_transform(doc_embeddings)

# Extract coordinates
x_coords = reduced_pca[:, 0]
y_coords = reduced_pca[:, 1]

# Create figure
plt.figure(figsize=(10, 7))
clusters = [doc['cluster_label'] for doc in documents]
scatter = plt.scatter(x_coords, y_coords, c=clusters, cmap='tab10', alpha=0.6)

plt.title("K-Means Clusters (PCA 2D Projection)")
plt.xlabel("Principal Component 1")
plt.ylabel("Principal Component 2")

# Highlight labeled files
for i, doc in enumerate(documents):
    filename = doc['normalized_filename']
    if filename in labeled_files_class0 or filename in labeled_files_class1:
        plt.scatter(x_coords[i], y_coords[i], c='none', edgecolors='black', marker='o', s=100, linewidths=1.5)

        # Label the points
        if filename in labeled_files_class0:
            label_text = "C0"
        elif filename in labeled_files_class1:
            label_text = "C1"
        plt.text(x_coords[i] + 0.02, y_coords[i] + 0.02, label_text, fontsize=9)

# Add legend for clusters
unique_clusters = list(sorted(set(clusters)))
handles = scatter.legend_elements()[0]
labels_for_legend = [f"Cluster {uc}" for uc in unique_clusters]
plt.legend(handles, labels_for_legend, title="Clusters", loc="best")

plt.show()

Step 3: Correct Cosine Similarity Visualization Code

Update the cosine similarity plot to ensure all labeled files are included:

# Extract cosine similarity values
similarities = [doc['similarity_to_nonpub'] for doc in documents]

# Categorize similarities
similarities_labeled_c0 = []
similarities_labeled_c1 = []

for doc in documents:
    filename = doc['normalized_filename']
    sim_val = doc['similarity_to_nonpub']
    if filename in labeled_files_class0:
        similarities_labeled_c0.append(sim_val)
    elif filename in labeled_files_class1:
        similarities_labeled_c1.append(sim_val)

plt.figure(figsize=(10, 6))

# Plot overall distribution
sns.histplot(similarities, kde=True, color='gray', alpha=0.4, label='All Documents')

# Highlight class 0
sns.histplot(similarities_labeled_c0, kde=True, color='red', alpha=0.8, label='Class 0 (R001-R005)')

# Highlight class 1
sns.histplot(similarities_labeled_c1, kde=True, color='blue', alpha=0.6, label='Class 1 (R006-R015)')

plt.title("Cosine Similarity to Non-Publishable Centroid")
plt.xlabel("Similarity Score")
plt.ylabel("Count")
plt.legend(loc='best')

plt.show()

Step 4: Debugging Mismatches

If any of the 15 labeled files still fail to appear in the plots, print the filenames for debugging:

# Check which labeled files are not being matched
missing_files_c0 = labeled_files_class0 - {doc['normalized_filename'] for doc in documents}
missing_files_c1 = labeled_files_class1 - {doc['normalized_filename'] for doc in documents}

print("Missing Class 0 Files:", missing_files_c0)
print("Missing Class 1 Files:", missing_files_c1)

Ensure the filenames in the labeled lists (labeled_files_class0, labeled_files_class1) exactly match the filenames in documents.

Explanation of Changes

1. Normalized Filenames: Ensured consistency by normalizing all filenames to lowercase and stripping whitespace before matching.
2. Debugging Missing Matches: Added code to detect and print any labeled files that aren’t matched.
3. Updated Visualization: Plots now handle all 15 labeled files and highlight them appropriately.

Let me know if the updated code resolves the issue or if further adjustments are needed!

Here's how you can adjust the previous visualization codes to highlight the documents where maybe_nonpub == True. This involves updating both the clustering visualization and the cosine similarity distribution visualization.

1. Clustering Visualization with maybe_nonpub Highlighting

We’ll add a special highlight for documents marked as maybe_nonpub == True in the clustering plot. These will be shown with a distinct edge color (e.g., orange).

# PCA for dimensionality reduction
pca = PCA(n_components=2, random_state=42)
reduced_pca = pca.fit_transform(doc_embeddings)

# Extract coordinates
x_coords = reduced_pca[:, 0]
y_coords = reduced_pca[:, 1]

# Create figure
plt.figure(figsize=(10, 7))
clusters = [doc['cluster_label'] for doc in documents]
scatter = plt.scatter(x_coords, y_coords, c=clusters, cmap='tab10', alpha=0.6)

plt.title("K-Means Clusters (PCA 2D Projection)")
plt.xlabel("Principal Component 1")
plt.ylabel("Principal Component 2")

# Highlight labeled files
for i, doc in enumerate(documents):
    filename = doc['normalized_filename']
    # Highlight labeled documents (Class 0 and Class 1)
    if filename in labeled_files_class0 or filename in labeled_files_class1:
        plt.scatter(x_coords[i], y_coords[i], c='none', edgecolors='black', marker='o', s=100, linewidths=1.5)
        if filename in labeled_files_class0:
            label_text = "C0"
        elif filename in labeled_files_class1:
            label_text = "C1"
        plt.text(x_coords[i] + 0.02, y_coords[i] + 0.02, label_text, fontsize=9)

    # Highlight maybe_nonpub documents with a distinct edge color
    if doc.get('maybe_nonpub', False):
        plt.scatter(x_coords[i], y_coords[i], c='none', edgecolors='orange', marker='s', s=120, linewidths=1.5)

# Add legend for clusters
unique_clusters = list(sorted(set(clusters)))
handles = scatter.legend_elements()[0]
labels_for_legend = [f"Cluster {uc}" for uc in unique_clusters]
plt.legend(handles, labels_for_legend, title="Clusters", loc="best")

plt.show()

Explanation:

1. Black Circles: Highlight the labeled documents (R001 to R015) with class annotations (C0 and C1).
2. Orange Squares: Highlight documents where maybe_nonpub == True with orange square edges.

2. Cosine Similarity Distribution with maybe_nonpub Highlighting

We’ll adjust the cosine similarity plot to highlight the distribution of documents where maybe_nonpub == True in addition to the labeled documents.

# Extract cosine similarity values
similarities = [doc['similarity_to_nonpub'] for doc in documents]

# Categorize similarities
similarities_labeled_c0 = []
similarities_labeled_c1 = []
similarities_maybe_nonpub = []

for doc in documents:
    filename = doc['normalized_filename']
    sim_val = doc['similarity_to_nonpub']
    if filename in labeled_files_class0:
        similarities_labeled_c0.append(sim_val)
    elif filename in labeled_files_class1:
        similarities_labeled_c1.append(sim_val)
    if doc.get('maybe_nonpub', False):
        similarities_maybe_nonpub.append(sim_val)

plt.figure(figsize=(10, 6))

# Plot overall distribution
sns.histplot(similarities, kde=True, color='gray', alpha=0.4, label='All Documents')

# Highlight class 0
sns.histplot(similarities_labeled_c0, kde=True, color='red', alpha=0.8, label='Class 0 (R001-R005)')

# Highlight class 1
sns.histplot(similarities_labeled_c1, kde=True, color='blue', alpha=0.6, label='Class 1 (R006-R015)')

# Highlight maybe_nonpub
sns.histplot(similarities_maybe_nonpub, kde=True, color='orange', alpha=0.5, label='Maybe Non-Publishable')

plt.title("Cosine Similarity to Non-Publishable Centroid")
plt.xlabel("Similarity Score")
plt.ylabel("Count")
plt.legend(loc='best')

plt.show()

Explanation:

1. Gray Distribution: All documents.
2. Red and Blue Distributions: Labeled Class 0 (R001-R005) and Class 1 (R006-R015), respectively.
3. Orange Distribution: Documents marked as maybe_nonpub == True.

Testing and Debugging

To ensure the plots work without errors:

1. Check that every document in documents has a normalized filename (normalized_filename) and the maybe_nonpub attribute (default to False if not already present).
2. If errors persist, print out examples of documents missing expected attributes:

   for doc in documents:
       if 'normalized_filename' not in doc or 'maybe_nonpub' not in doc:
           print("Missing attributes in:", doc)
   

Let me know if the updated code highlights the maybe_nonpub documents correctly and if further refinements are needed!

Thought for 7 seconds

Below is a detailed answer addressing your specific questions:

1. Is it viable to iteratively update the non-publishable centroid by adding newly identified “high-similarity” papers to the known non-publishable list?
2. Why might a small MLP (or logistic regression) classifier be useful, and how do we integrate it with the existing code?
3. Should we replace K-Means with something like Hierarchical Clustering or HDBSCAN, and will that likely improve performance?

Finally, I provide two complete code examples (so you can simply paste them into your workflow):

• Set 1: The first code keeps your K-Means + centroid-based similarity approach, adds an iterative centroid expansion (semi-supervised “self-labeling”) step, and then demonstrates a small MLP for classification, complementing the similarity approach.
• Set 2: The second code tries HDBSCAN for outlier/cluster discovery instead of K-Means, and also shows an alternative logistic regression or MLP approach—just enough to illustrate how you might integrate another clustering approach and a classifier.

1. Iterative Centroid Expansion: Pros and Cons

Pros

1. Simple Semi-Supervised Method: By adding unlabeled documents that have high similarity to the known “non-publishable” centroid, you effectively enlarge your labeled “non-publishable” pool.
2. Refined Centroid: Each iteration recalculates the centroid with the newly added documents, potentially capturing more nuanced aspects of “non-publishable” text.
3. Fewer False Negatives: If your initial set is correct, you are less likely to miss documents that are truly “non-publishable.”

Cons

1. Risk of Error Propagation: If a paper is mistakenly added to the “non-publishable” pool, it can shift the centroid incorrectly and cause more incorrect classifications.
2. Subjectivity of Similarity Threshold: You must set a similarity threshold. If it’s too low, you’ll add documents too eagerly; if it’s too high, you might ignore borderline papers.

Practical Tip: Start with a higher threshold (e.g., > 0.60) to keep false positives low. Then you can slowly lower the threshold if you want to catch borderline cases, but watch for error drift.

2. Small MLP or Logistic Regression

Why Might This Help?

1. Combining Multiple Features: A simple centroid-based similarity is only one dimension. If you also incorporate features such as “embedding vectors,” “similarity_to_nonpub,” or “red_flag_score,” your classifier can learn a more robust boundary.
2. Cross-Validation: Even with 15 labeled examples, you can do cross-validation (like 5-fold or leave-one-out) to help prevent overfitting and select hyperparameters.
3. Capture Non-Linear Boundaries: An MLP can model more complex decision surfaces than a linear threshold on similarity.

Downsides

1. Risk of Overfitting: 15 labeled examples is very few.
2. You Must Carefully Validate: Rely on cross-validation or you won’t know if you’re just memorizing the 15 examples.

3. Alternatives to K-Means: HDBSCAN or Hierarchical

• HDBSCAN: A density-based clustering algorithm that can identify outliers as their own cluster or leave them as unassigned. This might catch unusual documents that do not fit into your main “publishable” clusters.
• Hierarchical Clustering: Can give you a dendrogram and you can decide the cut. But it can be memory-intensive for large datasets.
• Will It Improve Performance? Possibly, if your data truly forms irregular clusters or if you care about identifying “noise/outlier” documents specifically. However, K-Means is simpler and often works well for dense embeddings. It’s worth an experiment if you have time.

Set 1: End-to-End Code (Iterative Centroid + K-Means + MLP)

Below is a consolidated code block illustrating:

1. Iterative centroid expansion for “non-publishable” detection.
2. An MLP that uses the paper embeddings plus similarity_to_nonpub as input features.

Where to paste: Ideally, after you’ve computed doc_embeddings, similarity_to_nonpub, and maybe_nonpub for each document. This can replace or supplement the final classification steps.

###############################################################################
# SET 1: Iterative Centroid Expansion + K-Means + MLP
###############################################################################

import numpy as np
from copy import deepcopy
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import cross_val_score
import warnings
warnings.filterwarnings('ignore')

###############################################################################
# 1. Iterative Centroid Expansion
###############################################################################

def iterative_nonpub_expansion(documents, known_nonpub_filenames, similarity_threshold=0.60, max_iterations=5):
    """
    Iteratively expands the known non-publishable list by adding any documents
    with similarity > similarity_threshold. Then recalculates centroid.
    
    Arguments:
    - documents: list of dicts containing doc['filename'], doc['similarity_to_nonpub'], etc.
    - known_nonpub_filenames: initial set/list of known non-publishable filenames (strings).
    - similarity_threshold: similarity above which a doc is considered "non-publishable."
    - max_iterations: maximum number of expansions to run.
    
    Returns:
    - updated_nonpub_filenames: final set of non-publishable filenames after expansion
    """
    updated_nonpub_filenames = set(known_nonpub_filenames)
    iteration = 0

    while iteration < max_iterations:
        iteration += 1
        # Calculate new centroid from the current set
        nonpub_embeddings = []
        for doc in documents:
            if doc['filename'] in updated_nonpub_filenames:
                emb = doc['doc_embedding']  # we store doc_embedding for convenience
                nonpub_embeddings.append(emb)
        new_centroid = np.mean(nonpub_embeddings, axis=0)

        # Add docs that exceed the threshold
        newly_added = False
        for doc in documents:
            if doc['filename'] not in updated_nonpub_filenames:
                # compute similarity to new centroid
                emb = doc['doc_embedding']
                sim = cosine_similarity(emb, new_centroid)
                if sim > similarity_threshold:
                    updated_nonpub_filenames.add(doc['filename'])
                    newly_added = True
        
        if not newly_added:
            print(f"No new docs added in iteration {iteration}. Stopping.")
            break
        else:
            print(f"Iteration {iteration} added new docs. Current size of non-pub set: {len(updated_nonpub_filenames)}.")

    return updated_nonpub_filenames

# We'll assume you already have doc['doc_embedding'] = get_document_embedding(...)
# To store doc_embedding in your existing pipeline, do something like:
# for i, doc in enumerate(documents):
#     doc['doc_embedding'] = doc_embeddings[i]

# 1.1: Store embeddings in each doc for convenience
for i, doc in enumerate(documents):
    doc['doc_embedding'] = doc_embeddings[i]

# 1.2: Perform initial similarity calculation (already done in your code, typically)
# for doc in documents:
#     sim = cosine_similarity(doc['doc_embedding'], nonpub_centroid)
#     doc['similarity_to_nonpub'] = float(sim)

# 1.3: Iteratively expand
known_nonpub_initial = ["R001.pdf", "R002.pdf", "R003.pdf", "R004.pdf", "R005.pdf"]
expanded_nonpub_filenames = iterative_nonpub_expansion(documents, known_nonpub_initial,
                                                       similarity_threshold=0.60,
                                                       max_iterations=5)
print("Final set of non-publishable (expanded) = ", expanded_nonpub_filenames)


###############################################################################
# 2. K-Means (Already in your code)
###############################################################################
# We'll keep your existing KMeans step. No changes needed unless you want to integrate
# the newly expanded non-publishable set in some way (like removing them from K-Means, etc.).
# 
# For reference:
# labels = kmeans.fit_predict(doc_embeddings)
# for i, doc in enumerate(documents):
#     doc['cluster_label'] = int(labels[i])


###############################################################################
# 3. Small MLP for classification
###############################################################################
# We'll build a tiny dataset from the 15 labeled documents you have 
# (including your newly expanded nonpub set if you like). 
# Then we do cross-validation to evaluate how well it classifies 
# publishable (class 1) vs non-publishable (class 0).

from sklearn.model_selection import StratifiedKFold

# Let's define your 15 labeled docs (or any subset you trust):
nonpub_set_c0 = set(["R001.pdf", "R002.pdf", "R003.pdf", "R004.pdf", "R005.pdf"])
pub_set_c1 = set(["R006.pdf", "R007.pdf", "R008.pdf", "R009.pdf", "R010.pdf",
                  "R011.pdf", "R012.pdf", "R013.pdf", "R014.pdf", "R015.pdf"])

X = []
y = []

for doc in documents:
    fname = doc['filename']
    if fname in nonpub_set_c0:
        X.append(doc['doc_embedding'])
        y.append(0)
    elif fname in pub_set_c1:
        X.append(doc['doc_embedding'])
        y.append(1)

X = np.array(X)
y = np.array(y)

# If we want to include similarity as an extra feature:
# We'll combine doc_embedding with similarity_to_nonpub
# Just do a quick horizontal stack: 
X_aug = []
for doc, label in zip(documents, range(len(documents))):
    # but we only want those that are in our labeled set
    fname = doc['filename']
    if fname in nonpub_set_c0 or fname in pub_set_c1:
        embed_768 = doc['doc_embedding']
        sim_val = doc['similarity_to_nonpub']  # a single float
        # create a feature vector of size 769
        new_vec = np.concatenate([embed_768, [sim_val]])
        X_aug.append(new_vec)

# We must track y in the same order
y_aug = []
for doc in documents:
    fname = doc['filename']
    if fname in nonpub_set_c0:
        y_aug.append(0)
    elif fname in pub_set_c1:
        y_aug.append(1)

X_aug = np.array(X_aug)
y_aug = np.array(y_aug)

# We'll train an MLP on X_aug, y_aug
mlp = MLPClassifier(hidden_layer_sizes=(64,), random_state=42, max_iter=1000)

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(mlp, X_aug, y_aug, cv=skf, scoring='accuracy')
print(f"5-Fold CV Accuracy for MLP: {scores.mean():.2f} +/- {scores.std():.2f}")

mlp.fit(X_aug, y_aug)

# Now we can apply this MLP to all 150 documents
# They each get a predicted label: 0 = non-publishable, 1 = publishable
for doc in documents:
    embed_768 = doc['doc_embedding']
    sim_val = doc['similarity_to_nonpub']
    test_vec = np.concatenate([embed_768, [sim_val]])
    pred_label = mlp.predict(test_vec.reshape(1, -1))[0]
    doc['mlp_pred_label'] = int(pred_label)

# doc['mlp_pred_label'] == 0 => non-publishable, 1 => publishable

Where to Paste This in Your Code

• Step 1 (Iterative expansion) should come after you’ve computed doc_embeddings and the initial nonpub_centroid.
• Step 2 is just your existing K-Means code—no big change required.
• Step 3 (MLP) can go at the very end, after you have doc_embeddings and doc['similarity_to_nonpub'].

Set 2: Alternative Clustering (HDBSCAN) + (Optionally) MLP

This second code snippet replaces K-Means with HDBSCAN or Hierarchical Clustering. HDBSCAN requires installing the library:

!pip install hdbscan

We’ll also show how to do the MLP in the same manner. (You can remove the MLP part if you only want to test clustering.)

Where to paste: Replace your K-Means block with the HDBSCAN block. Keep the rest of your pipeline the same.

###############################################################################
# SET 2: Using HDBSCAN (Outlier Detection) + Optional MLP
###############################################################################
!pip install hdbscan
import hdbscan

# 1) HDBSCAN to cluster doc_embeddings
hdb = hdbscan.HDBSCAN(min_cluster_size=5, min_samples=3, metric='euclidean')
hdb_labels = hdb.fit_predict(doc_embeddings)

# We can store these cluster labels in documents
for i, doc in enumerate(documents):
    doc['hdb_cluster_label'] = int(hdb_labels[i])  # -1 means outlier/no cluster

# Let's see how many are labeled as outliers
outlier_count = sum(1 for lbl in hdb_labels if lbl == -1)
print(f"Number of outliers (no cluster assigned): {outlier_count}")

# 2) Inspect cluster distribution
label_counts = Counter(hdb_labels)
print("HDBSCAN Cluster Distribution:", label_counts)

# 3) If you want a quick visualization:
import matplotlib.pyplot as plt
import numpy as np
from sklearn.decomposition import PCA

pca = PCA(n_components=2, random_state=42)
reduced_pca = pca.fit_transform(doc_embeddings)

x_coords = reduced_pca[:,0]
y_coords = reduced_pca[:,1]

plt.figure(figsize=(10,7))
scatter = plt.scatter(x_coords, y_coords, c=hdb_labels, cmap='tab10', alpha=0.6)
plt.title("HDBSCAN Clusters (PCA 2D Projection)")
plt.xlabel("PC1")
plt.ylabel("PC2")

# highlight your known labeled docs
labeled_files_class0 = set(["R001.pdf", "R002.pdf", "R003.pdf", "R004.pdf", "R005.pdf"])
labeled_files_class1 = set(["R006.pdf", "R007.pdf", "R008.pdf", "R009.pdf", "R010.pdf",
                            "R011.pdf", "R012.pdf", "R013.pdf", "R014.pdf", "R015.pdf"])

for i, doc in enumerate(documents):
    if doc['filename'] in labeled_files_class0 or doc['filename'] in labeled_files_class1:
        plt.scatter(x_coords[i], y_coords[i], c='none', edgecolors='black',
                    marker='o', s=100, linewidths=1.5)
        if doc['filename'] in labeled_files_class0:
            label_text = "C0"
        else:
            label_text = "C1"
        plt.text(x_coords[i] + 0.02, y_coords[i] + 0.02, label_text, fontsize=9)

plt.show()

# 4) Optional: Train an MLP or logistic regression using same approach as above
# Build labeled X, y from the 15 known docs
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import StratifiedKFold

X_aug = []
y_aug = []

for doc in documents:
    fname = doc['filename']
    if fname in labeled_files_class0:
        # Create feature vector from doc['doc_embedding'] + doc['similarity_to_nonpub']
        embed_768 = doc_embeddings[[i for i, d in enumerate(documents) if d['filename'] == fname][0]]
        sim_val = doc['similarity_to_nonpub']
        new_vec = np.concatenate([embed_768, [sim_val]])
        X_aug.append(new_vec)
        y_aug.append(0)
    elif fname in labeled_files_class1:
        embed_768 = doc_embeddings[[i for i, d in enumerate(documents) if d['filename'] == fname][0]]
        sim_val = doc['similarity_to_nonpub']
        new_vec = np.concatenate([embed_768, [sim_val]])
        X_aug.append(new_vec)
        y_aug.append(1)

X_aug = np.array(X_aug)
y_aug = np.array(y_aug)

mlp = MLPClassifier(hidden_layer_sizes=(64,), random_state=42, max_iter=1000)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(mlp, X_aug, y_aug, cv=skf, scoring='accuracy')
print(f"[HDBSCAN] 5-Fold CV Accuracy for MLP: {scores.mean():.2f} +/- {scores.std():.2f}")

mlp.fit(X_aug, y_aug)

for doc in documents:
    # For each doc, build the same feature vector and predict
    embed_768 = doc['doc_embedding']
    sim_val = doc['similarity_to_nonpub']
    test_vec = np.concatenate([embed_768, [sim_val]])
    pred_label = mlp.predict(test_vec.reshape(1, -1))[0]
    doc['mlp_pred_label'] = int(pred_label)

print("Done: HDBSCAN + MLP classification.")