【机器学习】AAAI 会议论文聚类分析

实验五:AAAI 会议论文聚类分析

​ 本次实验以AAAI 2014会议论文数据为基础,要求实现或调用无监督聚类算法,了解聚类方法。

1 任务介绍

​ 每年国际上召开的大大小小学术会议不计其数,发表了非常多的论文。在计算机领域的一些大型学术会议上,一次就可以发表涉及各个方向的几百篇论文。按论文的主题、内容进行聚类,有助于人们高效地查找和获得所需要的论文。本案例数据来源于AAAI 2014上发表的约400篇文章,由UCI公开提供,提供包括标题、作者、关键词、摘要在内的信息,希望大家能根据这些信息,合理地构造特征向量来表示这些论文,并设计实现或调用聚类算法对论文进行聚类。最后也可以对聚类结果进行观察,看每一类都是什么样的论文,是否有一些主题。

1.1 基本要求:

  1. 将文本转化为向量,实现或调用无监督聚类算法,对论文聚类,例如10类(可使用已有工具包例如sklearn);
  2. 观察每一类中的论文,调整算法使结果较为合理;
  3. 无监督聚类没有标签,效果较难评价,因此没有硬性指标,跑通即可,主要让大家了解和感受聚类算法,比较简单。

1.2 扩展要求:

  1. 对文本向量进行降维,并将聚类结果可视化成散点图。

注:group和topic也不能完全算是标签,因为

  1. 有些文章作者投稿时可能会选择某个group/topic但实际和另外group/topic也相关甚至更相关;
  2. 一篇文章可能有多个group和topic,作为标签会出现有的文章同属多个类别,这里暂不考虑这样的聚类;
  3. group和topic的取值很多,但聚类常常希望指定聚合成出例如5/10/20类;
  4. 感兴趣但同学可以思考利用group和topic信息来量化评价无监督聚类结果,不作要求。

1.3 提示:

  1. 高维向量的降维旨在去除一些高相关性的特征维度,保留最有用的信息,用更低维的向量表示高维数据,常用的方法有PCA和t-SNE等;
  2. 降维与聚类是两件不同的事情,聚类实际上在降维前的高维向量和降维后的低维向量上都可以进行,结果也可能截然不同;
  3. 高维向量做聚类,降维可视化后若有同一类的点不在一起,是正常的。在高维空间中它们可能是在一起的,降维后损失了一些信息。
import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
import re
import nltk
import sklearn
import seaborn as sns # 作图
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from scipy import sparse # 稀疏矩阵RANDOM_STATE = 2023

2 导入数据

data_df = pd.read_csv('./data/[UCI] AAAI-14 Accepted Papers - Papers.csv') # 读入 csv 文件为 pandas 的 DataFrame
data_df.head(5) 
titleauthorsgroupskeywordstopicsabstract
0Kernelized Bayesian Transfer LearningMehmet Gönen and Adam A. MargolinNovel Machine Learning Algorithms (NMLA)cross-domain learning\ndomain adaptation\nkern...APP: Biomedical / Bioinformatics\nNMLA: Bayesi...Transfer learning considers related but distin...
1"Source Free" Transfer Learning for Text Class...Zhongqi Lu, Yin Zhu, Sinno Pan, Evan Xiang, Yu...AI and the Web (AIW)\nNovel Machine Learning A...Transfer Learning\nAuxiliary Data Retrieval\nT...AIW: Knowledge acquisition from the web\nAIW: ...Transfer learning uses relevant auxiliary data...
2A Generalization of Probabilistic Serial to Ra...Haris Aziz and Paul StursbergGame Theory and Economic Paradigms (GTEP)social choice theory\nvoting\nfair division\ns...GTEP: Game Theory\nGTEP: Social Choice / VotingThe probabilistic serial (PS) rule is one of t...
3Lifetime Lexical Variation in Social MediaLiao Lizi, Jing Jiang, Ying Ding, Heyan Huang ...NLP and Text Mining (NLPTM)Generative model\nSocial Networks\nAge PredictionAIW: Web personalization and user modeling\nNL...As the rapid growth of online social media att...
4Hybrid Singular Value Thresholding for Tensor ...Xiaoqin Zhang, Zhengyuan Zhou, Di Wang and Yi MaKnowledge Representation and Reasoning (KRR)\n...tensor completion\nlow-rank recovery\nhybrid s...KRR: Knowledge Representation (General/Other)\...In this paper, we study the low-rank tensor co...

查看dataframe数据信息:

data_df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 398 entries, 0 to 397
Data columns (total 6 columns):#   Column    Non-Null Count  Dtype 
---  ------    --------------  ----- 0   title     398 non-null    object1   authors   398 non-null    object2   groups    396 non-null    object3   keywords  398 non-null    object4   topics    394 non-null    object5   abstract  398 non-null    object
dtypes: object(6)
memory usage: 18.8+ KB

从以上信息可以看出,data_df存在空数据,应对其作处理

# stack()将df转换为series对象; [lambda x:x]只保留True元素
data_df.isnull().stack()[lambda x: x]
211  groups    True
340  groups    True
344  topics    True
364  topics    True
365  topics    True
388  topics    True
dtype: bool

对空数据进行填充为空字符处理

data_df = data_df.fillna('') # 填充空值为空字符串

3 文本想量化

3.1 简单文本向量化

将同一篇文章的不同类型数据结合,选择使用TF-IDF模型,对文本进行向量化

paper_df = data_df['title']+' '+data_df['authors']+' '+data_df['groups']+' '\
+data_df['keywords']+' '+data_df['topics']+' '+data_df['abstract']paper_df

结果:

0      Kernelized Bayesian Transfer Learning Mehmet G...
1      "Source Free" Transfer Learning for Text Class...
2      A Generalization of Probabilistic Serial to Ra...
3      Lifetime Lexical Variation in Social Media Lia...
4      Hybrid Singular Value Thresholding for Tensor ......                        
393    Mapping Users Across Networks by Manifold Alig...
394    Compact Aspect Embedding For Diversified Query...
395    Contraction and Revision over DL-Lite TBoxes Z...
396    Zero Pronoun Resolution as Ranking Chen Chen a...
397    Supervised Transfer Sparse Coding Maruan Al-Sh...
Length: 398, dtype: object
vectorizer = TfidfVectorizer(max_df=0.9, min_df=10)
X_simple = vectorizer.fit_transform(paper_df)

3.2 复杂文本向量化

  1. 将作者名字分割合适
def author_tokenizer(text): authors = re.split("\s+and\s+|\s*,\s*", text) # 根据逗号或者and进行分词return authorsauthors = data_df['authors'][1]
author_split = author_tokenizer(authors)
print(authors,'\n',author_split)

结果:

Zhongqi Lu, Yin Zhu, Sinno Pan, Evan Xiang, Yujing Wang and Qiang Yang ['Zhongqi Lu', 'Yin Zhu', 'Sinno Pan', 'Evan Xiang', 'Yujing Wang', 'Qiang Yang']
  1. 将其他文本进行分词、去除停用词、词干化处理
def text_tokenizer(text):# 分词words = nltk.tokenize.word_tokenize(text)# 去除停用词stop_words = set(nltk.corpus.stopwords.words('english'))words = [word for word in words if word.lower() not in stop_words]# 词干化stemmer = nltk.stem.PorterStemmer()words = [stemmer.stem(word) for word in words]return wordsabstracts=data_df['abstract'][1]
abstracts_split = text_tokenizer(abstracts)
print(abstracts,'\n',abstracts_split)

结果:

Transfer learning uses relevant auxiliary data to help the learning task in a target domain where labeled data are usually insufficient to train an accurate model. Given appropriate auxiliary data, researchers have proposed many transfer learning models. How to find such auxiliary data, however, is of little research in the past. In this paper, we focus on this auxiliary data retrieval problem, and propose a transfer learning framework that effectively selects helpful auxiliary data from an open knowledge space (e.g. the World Wide Web). Because there is no need of manually selecting auxiliary data for different target domain tasks, we call our framework Source Free Transfer Learning (SFTL). For each target domain task, SFTL framework iteratively queries for the helpful auxiliary data based on the learned model and then updates the model using the retrieved auxiliary data. We highlight the automatic constructions of queries and the robustness of the SFTL framework. Our experiments on the 20 NewsGroup dataset and the Google search snippets dataset suggest that the new framework is capable to have the comparable performance to those state-of-the-art methods with dedicated selections of auxiliary data. ['transfer', 'learn', 'use', 'relev', 'auxiliari', 'data', 'help', 'learn', 'task', 'target', 'domain', 'label', 'data', 'usual', 'insuffici', 'train', 'accur', 'model', '.', 'given', 'appropri', 'auxiliari', 'data', ',', 'research', 'propos', 'mani', 'transfer', 'learn', 'model', '.', 'find', 'auxiliari', 'data', ',', 'howev', ',', 'littl', 'research', 'past', '.', 'paper', ',', 'focu', 'auxiliari', 'data', 'retriev', 'problem', ',', 'propos', 'transfer', 'learn', 'framework', 'effect', 'select', 'help', 'auxiliari', 'data', 'open', 'knowledg', 'space', '(', 'e.g', '.', 'world', 'wide', 'web', ')', '.', 'need', 'manual', 'select', 'auxiliari', 'data', 'differ', 'target', 'domain', 'task', ',', 'call', 'framework', 'sourc', 'free', 'transfer', 'learn', '(', 'sftl', ')', '.', 'target', 'domain', 'task', ',', 'sftl', 'framework', 'iter', 'queri', 'help', 'auxiliari', 'data', 'base', 'learn', 'model', 'updat', 'model', 'use', 'retriev', 'auxiliari', 'data', '.', 'highlight', 'automat', 'construct', 'queri', 'robust', 'sftl', 'framework', '.', 'experi', '20', 'newsgroup', 'dataset', 'googl', 'search', 'snippet', 'dataset', 'suggest', 'new', 'framework', 'capabl', 'compar', 'perform', 'state-of-the-art', 'method', 'dedic', 'select', 'auxiliari', 'data', '.']

查看每列名称:

data_df.columns

结果:

Index(['title', 'authors', 'groups', 'keywords', 'topics', 'abstract'], dtype='object')

创建 TF-IDF 矩阵:

vectorizer_authour = TfidfVectorizer(tokenizer = author_tokenizer)
vectorizer_text = TfidfVectorizer(tokenizer = text_tokenizer)
X_authours = vectorizer_authour.fit_transform(data_df['authors'].tolist()) 
X_title = vectorizer_text.fit_transform(data_df['title'].tolist()) 
X_groups = vectorizer_text.fit_transform(data_df['groups'].tolist()) 
X_keywords = vectorizer_text.fit_transform(data_df['keywords'].tolist()) 
X_topics = vectorizer_text.fit_transform(data_df['topics'].tolist()) vectorizer_texts = TfidfVectorizer(max_df=0.9, min_df=5, tokenizer = text_tokenizer)
X_abstract = vectorizer_texts.fit_transform(data_df['abstract'].tolist()) print(f'X_title:{X_title.shape}')
print(f'X_authours:{X_authours.shape}')
print(f'X_groups:{X_groups.shape}')
print(f'X_keywords:{X_keywords.shape}')
print(f'X_topics:{X_topics.shape}')
print(f'X_abstract:{X_abstract.shape}')

结果:

X_title:(398, 1124)
X_authours:(398, 1105)
X_groups:(398, 64)
X_keywords:(398, 1051)
X_topics:(398, 305)
X_abstract:(398, 1042)

将稀疏矩阵拼接

X_passage = sparse.hstack([X_title, X_authours, X_groups, X_keywords, X_topics, X_abstract]) # 稀疏向量拼接
print(X_passage.shape)
(398, 4691)

4 聚类算法

4.1 简单聚类

直接采用KMeans简单聚类

k = 5 #假设有5个类别
model = KMeans(n_clusters=k, init='k-means++', max_iter=100, n_init=1)
model.fit(X_simple)
labels = model.labels_
data_df['label'] = labels
labels

结果:

array([1, 1, 4, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 0, 1, 4, 1, 1, 1, 1, 3,2, 1, 0, 1, 1, 2, 1, 1, 4, 0, 1, 1, 4, 3, 1, 4, 1, 4, 1, 3, 1, 0,4, 3, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 3, 4, 1, 1, 4, 1, 3, 1, 1, 4,3, 4, 1, 3, 4, 2, 1, 1, 1, 1, 3, 4, 1, 4, 1, 1, 1, 1, 1, 1, 1, 4,1, 1, 1, 1, 0, 1, 1, 2, 1, 4, 1, 1, 3, 1, 1, 1, 3, 2, 4, 0, 1, 3,4, 2, 1, 3, 1, 2, 1, 4, 1, 1, 1, 1, 1, 0, 4, 1, 1, 0, 1, 0, 1, 3,1, 1, 4, 4, 1, 1, 0, 1, 3, 1, 1, 1, 1, 1, 0, 1, 0, 4, 1, 1, 0, 2,1, 2, 1, 0, 1, 1, 1, 4, 3, 1, 2, 1, 4, 3, 0, 2, 3, 4, 0, 3, 3, 1,1, 2, 4, 3, 3, 4, 1, 1, 3, 2, 1, 0, 4, 4, 4, 4, 2, 1, 1, 3, 0, 4,2, 1, 2, 0, 1, 1, 3, 3, 0, 1, 1, 1, 1, 1, 3, 1, 1, 1, 0, 1, 0, 1,1, 1, 1, 1, 4, 1, 3, 1, 1, 1, 3, 1, 1, 4, 1, 2, 3, 0, 2, 3, 1, 1,1, 1, 1, 4, 1, 0, 1, 1, 2, 1, 4, 1, 1, 1, 0, 1, 1, 1, 1, 4, 1, 1,1, 4, 0, 1, 1, 1, 4, 1, 4, 2, 1, 1, 1, 2, 1, 3, 1, 0, 1, 2, 2, 1,1, 3, 1, 1, 1, 3, 2, 1, 3, 4, 1, 1, 1, 1, 1, 1, 1, 4, 4, 1, 1, 4,0, 1, 1, 3, 0, 4, 2, 0, 1, 4, 1, 2, 4, 3, 1, 1, 3, 3, 3, 1, 1, 1,4, 1, 1, 2, 2, 1, 4, 4, 2, 1, 3, 0, 4, 4, 1, 0, 0, 4, 3, 1, 1, 1,3, 1, 3, 1, 3, 0, 1, 4, 1, 1, 1, 1, 1, 1, 4, 1, 1, 1, 4, 2, 1, 1,3, 1, 3, 0, 1, 1, 0, 1, 1, 3, 1, 1, 2, 2, 1, 2, 4, 0, 1, 1, 1, 3,1, 1])

总结分类规律

data_df[data_df['label']==4][['title', 'groups', 'topics']]
titlegroupstopics
2A Generalization of Probabilistic Serial to Ra...Game Theory and Economic Paradigms (GTEP)GTEP: Game Theory\nGTEP: Social Choice / Voting
16Multi-Organ Exchange: The Whole is Greater tha...Applications (APP)\nGame Theory and Economic P...APP: Biomedical / Bioinformatics\nGTEP: Auctio...
30The Computational Rise and Fall of FairnessGame Theory and Economic Paradigms (GTEP)GTEP: Social Choice / Voting
34Lazy Defenders Are Almost Optimal Against Dili...Game Theory and Economic Paradigms (GTEP)GTEP: Game Theory\nGTEP: Imperfect Information
37Game-theoretic Resource Allocation for Protect...Applications (APP)\nGame Theory and Economic P...APP: Security and Privacy\nGTEP: Game Theory\n...
39A Strategy-Proof Online Auction with Time Disc...Game Theory and Economic Paradigms (GTEP)GTEP: Auctions and Market-Based Systems
44Simultaneous Cake CuttingGame Theory and Economic Paradigms (GTEP)GTEP: Social Choice / Voting
57Solving Imperfect Information Games Using Deco...Game Theory and Economic Paradigms (GTEP)GTEP: Game Theory\nGTEP: Equilibrium\nGTEP: Im...
60Online (Budgeted) Social ChoiceGame Theory and Economic Paradigms (GTEP)GTEP: Game Theory\nGTEP: Social Choice / Votin...
65Fixing a Balanced Knockout TournamentGame Theory and Economic Paradigms (GTEP)GTEP: Game Theory\nGTEP: Social Choice / Voting
67Incomplete Preferences in Single-Peaked Electo...Game Theory and Economic Paradigms (GTEP)GTEP: Social Choice / Voting\nGTEP: Imperfect ...
70A Control Dichotomy for Pure Scoring RulesGame Theory and Economic Paradigms (GTEP)GTEP: Social Choice / Voting
77Biased GamesGame Theory and Economic Paradigms (GTEP)GTEP: Game Theory\nGTEP: Equilibrium
79Preference Elicitation and Interview Minimizat...Game Theory and Economic Paradigms (GTEP)\nMul...APP: Computational Social Science\nGTEP: Socia...
87Minimising Undesired Task Costs in Multi-robot...Multiagent Systems (MAS)\nRobotics (ROB)GTEP: Auctions and Market-Based Systems\nMAS: ...
97Congestion Games for V2G-Enabled EV ChargingComputational Sustainability and AI (CSAI)\nGa...CSAI: Modeling the interactions of agents with...
106Evolutionary dynamics of learning algorithms o...Game Theory and Economic Paradigms (GTEP)\nMul...GTEP: Adversarial Learning\nGTEP: Equilibrium\...
110A Game-theoretic Analysis of Catalog OptimizationGame Theory and Economic Paradigms (GTEP)\nKno...GTEP: Auctions and Market-Based Systems\nGTEP:...
117Automatic Game Design via Mechanic GenerationGame Playing and Interactive Entertainment (GPIE)GPIE: AI in Game Design\nGPIE: Procedural Cont...
124False-Name Bidding and Economic Efficiency in ...Game Theory and Economic Paradigms (GTEP)\nMul...GTEP: Auctions and Market-Based Systems\nGTEP:...
134Mechanism Design for Scheduling with Uncertain...Game Theory and Economic Paradigms (GTEP)\nMul...GTEP: Auctions and Market-Based Systems\nGTEP:...
135Robust Winners and Winner Determination Polici...Game Theory and Economic Paradigms (GTEP)\nMul...APP: Computational Social Science\nGTEP: Socia...
149Regret Transfer and Parameter OptimizationGame Theory and Economic Paradigms (GTEP)GTEP: Game Theory\nGTEP: Equilibrium\nGTEP: Im...
161Trading Multiple Indivisible Goods with Indiff...Game Theory and Economic Paradigms (GTEP)GTEP: Game Theory\nGTEP: Social Choice / Votin...
166Item Bidding for Combinatorial Public ProjectsGame Theory and Economic Paradigms (GTEP)\nMul...GTEP: Game Theory\nGTEP: Coordination and Coll...
171Increasing VCG revenue by decreasing the quali...Game Theory and Economic Paradigms (GTEP)\nMul...GTEP: Auctions and Market-Based Systems\nMAS: ...
178Theory of Cooperation in Complex Social NetworksGame Theory and Economic Paradigms (GTEP)\nMul...GTEP: Game Theory\nGTEP: Coordination and Coll...
181Prices Matter for the Parameterized Complexity...Game Theory and Economic Paradigms (GTEP)\nMul...GTEP: Game Theory\nGTEP: Social Choice / Votin...
188Incentives for Truthful Information Elicitatio...Game Theory and Economic Paradigms (GTEP)\nHum...GTEP: Game Theory\nGTEP: Equilibrium\nGTEP: Im...
189Equilibria in Epidemic Containment GamesApplications (APP)\nComputational Sustainabili...APP: Security and Privacy\nCSAI: Modeling the ...
190Beat the Cheater: Computing Game-Theoretic Str...Game Theory and Economic Paradigms (GTEP)\nMul...GTEP: Game Theory\nGTEP: Equilibrium\nGTEP: Im...
191A Characterization of the Single-Peaked Single...Game Theory and Economic Paradigms (GTEP)\nMul...GTEP: Game Theory\nGTEP: Social Choice / Votin...
197Efficient buyer groups for prediction-of-use e...Computational Sustainability and AI (CSAI)\nGa...CSAI: Modeling the interactions of agents with...
224On Detecting Nearly Structured Preference Prof...Game Theory and Economic Paradigms (GTEP)\nMul...GTEP: Social Choice / Voting
233Betting Strategies, Market Selection, and the ...Game Theory and Economic Paradigms (GTEP)GTEP: Auctions and Market-Based Systems
245Leveraging Fee-Based, Imperfect Advisors in Hu...Humans and AI (HAI)HAI: Human-Computer Interaction
252On the Structure of Synergies in Cooperative G...Game Theory and Economic Paradigms (GTEP)GTEP: Game Theory
261On the Incompatibility of Efficiency and Strat...Game Theory and Economic Paradigms (GTEP)GTEP: Game Theory\nGTEP: Social Choice / Voting
265Regret-based Optimization and Preference Elici...Game Theory and Economic Paradigms (GTEP)GTEP: Game Theory
270Modal Ranking: A Uniquely Robust Voting RuleGame Theory and Economic Paradigms (GTEP)GTEP: Social Choice / Voting
272Extending Tournament SolutionsGame Theory and Economic Paradigms (GTEP)GTEP: Social Choice / Voting
295On Computing Optimal Strategies in Open List P...Game Theory and Economic Paradigms (GTEP)\nMul...GTEP: Game Theory\nGTEP: Social Choice / Votin...
303Envy-Free Division of Sellable GoodsGame Theory and Economic Paradigms (GTEP)GTEP: Auctions and Market-Based Systems\nGTEP:...
304Potential-Aware Imperfect-Recall Abstraction w...Game Theory and Economic Paradigms (GTEP)GTEP: Game Theory\nGTEP: Imperfect Information
307Voting with Rank Dependent Scoring RulesGame Theory and Economic Paradigms (GTEP)GTEP: Auctions and Market-Based Systems\nGTEP:...
313Incentivizing High-quality Content from Hetero...Game Theory and Economic Paradigms (GTEP)\nMul...GTEP: Game Theory\nGTEP: Equilibrium\nGTEP: Im...
317New Models for Competitive ContagionGame Theory and Economic Paradigms (GTEP)GTEP: Game Theory\nGTEP: Equilibrium
320Approximate Equilibrium and Incentivizing Soci...Game Theory and Economic Paradigms (GTEP)GTEP: Game Theory\nGTEP: Coordination and Coll...
330Internally Stable Kidney ExchangeMultiagent Systems (MAS)GTEP: Auctions and Market-Based Systems\nMAS: ...
336Strategyproof exchange with multiple private e...Game Theory and Economic Paradigms (GTEP)GTEP: Auctions and Market-Based Systems\nGTEP:...
337Mechanism design for mobile geo-location adver...Game Theory and Economic Paradigms (GTEP)\nMul...GTEP: Auctions and Market-Based Systems\nGTEP:...
342A Multiarmed Bandit Incentive Mechanism for Cr...Computational Sustainability and AI (CSAI)\nGa...CSAI: Modeling the interactions of agents with...
343Binary Aggregation by Selection of the Most Re...Game Theory and Economic Paradigms (GTEP)\nKno...GTEP: Social Choice / Voting\nKRR: Preferences...
347Bounding the Support Size in Extensive Form Ga...Game Theory and Economic Paradigms (GTEP)GTEP: Game Theory\nGTEP: Equilibrium\nGTEP: Im...
359The Fisher Market Game: Equilibrium and WelfareGame Theory and Economic Paradigms (GTEP)GTEP: Auctions and Market-Based Systems\nGTEP:...
366On the Axiomatic Characterization of Runoff Vo...Game Theory and Economic Paradigms (GTEP)GTEP: Social Choice / Voting
370Solving Zero-Sum Security Games in Discretized...Game Theory and Economic Paradigms (GTEP)\nMul...GTEP: Game Theory\nGTEP: Equilibrium\nMAS: Mul...
390Using Response Functions to Measure Strategy S...Game Theory and Economic Paradigms (GTEP)GTEP: Game Theory\nGTEP: Equilibrium\nGTEP: Im...

通过查看每组聚类结果得知:

  • 0:该类主要包含 VIS 计算机视觉 等文章
  • 1:该类主要包含 AIW 及网络类文章
  • 2:该类主要包含 NMLA 及算法类文章
  • 3:该类主要包含 GTEP 等游戏类文章
  • 4:该类主要包含 AIW 及NLP等文章

通过上述结果可知,简单聚类可以将文章分为几类,但是相互有所粘连

# 创建一个TSNE对象,指定要降维到的维数为2,随机数种子为RANDOM_STATE
tsne = sklearn.manifold.TSNE(n_components=2, random_state=RANDOM_STATE, init="random")# 调用TSNE对象的fit_transform方法,传入X_simple数据集,返回一个降维后的数据数组,赋值给X_tsne
X_tsne = tsne.fit_transform(X_simple)sns.scatterplot(x=X_tsne[:,0], y=X_tsne[:,1], hue=labels, palette="deep") # 散点图


请添加图片描述

通过上图显示,简单聚类可以成功聚类,但是结果有所粘连

4.2 复杂聚类

通过使用3.2中得到的X_pasage进行聚类,并聚集10类

model = KMeans(n_clusters=10,  init='k-means++', max_iter=100, n_init=1, random_state=RANDOM_STATE) # KMean聚类
model.fit(X_passage)
labels = model.labels_
data_df['label'] = labels
labels
array([2, 4, 3, 4, 0, 0, 8, 2, 4, 0, 5, 6, 2, 4, 6, 8, 3, 4, 4, 0, 1, 9,3, 2, 2, 8, 4, 5, 4, 8, 3, 5, 2, 2, 7, 9, 2, 7, 8, 7, 2, 1, 2, 6,3, 9, 2, 4, 4, 5, 4, 4, 4, 4, 2, 1, 9, 7, 1, 2, 3, 2, 4, 8, 4, 3,9, 3, 8, 9, 3, 9, 2, 2, 8, 8, 9, 7, 8, 3, 1, 4, 4, 0, 8, 1, 2, 3,8, 2, 2, 4, 6, 1, 2, 5, 0, 7, 8, 4, 9, 4, 2, 4, 6, 5, 7, 6, 8, 9,7, 5, 0, 9, 3, 5, 4, 7, 2, 0, 4, 2, 6, 6, 3, 8, 2, 6, 2, 9, 2, 1,6, 2, 3, 3, 8, 0, 6, 4, 9, 2, 6, 4, 2, 8, 5, 2, 6, 7, 2, 0, 6, 3,2, 5, 4, 6, 2, 4, 2, 3, 9, 2, 5, 8, 3, 1, 9, 3, 9, 3, 6, 9, 9, 2,8, 5, 3, 9, 9, 3, 8, 0, 1, 5, 4, 5, 3, 7, 7, 3, 5, 2, 2, 6, 6, 3,5, 2, 5, 6, 8, 4, 4, 9, 6, 8, 0, 1, 2, 2, 6, 4, 8, 4, 6, 2, 6, 0,4, 2, 8, 1, 3, 2, 4, 4, 4, 8, 6, 8, 2, 7, 4, 5, 3, 6, 5, 8, 2, 4,2, 2, 4, 4, 8, 5, 2, 2, 5, 0, 7, 2, 2, 4, 5, 0, 2, 2, 2, 3, 4, 0,2, 7, 5, 4, 1, 4, 3, 2, 3, 5, 2, 2, 8, 5, 2, 9, 4, 6, 2, 5, 5, 0,2, 9, 2, 4, 1, 9, 5, 2, 9, 3, 9, 4, 2, 2, 2, 8, 2, 3, 7, 8, 4, 3,9, 8, 5, 1, 6, 7, 5, 6, 2, 7, 2, 5, 7, 9, 8, 6, 9, 1, 9, 2, 2, 2,3, 2, 8, 5, 5, 8, 7, 3, 5, 8, 1, 6, 7, 3, 8, 6, 6, 7, 9, 0, 0, 4,9, 2, 1, 2, 9, 6, 2, 7, 8, 4, 6, 8, 2, 2, 3, 4, 1, 0, 7, 5, 2, 8,9, 4, 1, 6, 2, 8, 6, 0, 9, 9, 6, 4, 5, 5, 2, 5, 7, 6, 2, 1, 4, 9,4, 2])
data_df[data_df['label']==9][['title', 'groups', 'topics']]
titlegroupstopics
21The Complexity of Reasoning with FODD and GFODDKnowledge Representation and Reasoning (KRR)KRR: Automated Reasoning and Theorem Proving\n...
35PREGO: An Action Language for Belief-Based Cog...Knowledge Representation and Reasoning (KRR)KRR: Action, Change, and Causality\nKRR: Knowl...
45Recovering from Selection Bias in Causal and S...Knowledge Representation and Reasoning (KRR)\n...KRR: Action, Change, and Causality\nRU: Bayesi...
56A Parameterized Complexity Analysis of General...Game Playing and Interactive Entertainment (GP...GTEP: Social Choice / Voting\nKRR: Computation...
66Querying Inconsistent Description Logic Knowle...Knowledge Representation and Reasoning (KRR)KRR: Ontologies\nKRR: Computational Complexity...
69Knowledge Graph Embedding by Translating on Hy...Knowledge Representation and Reasoning (KRR)\n...KRR: Knowledge Representation (General/Other)\...
71Fast consistency checking of very large real-w...Knowledge Representation and Reasoning (KRR)\n...KRR: Geometric, Spatial, and Temporal Reasonin...
76The Computational Complexity of Structure-Base...Knowledge Representation and Reasoning (KRR)\n...KRR: Action, Change, and Causality\nKRR: Compu...
100A Tractable Approach to ABox Abduction over De...Knowledge Representation and Reasoning (KRR)KRR: Description Logics\nKRR: Diagnosis and Ab...
109Reasoning on LTL on Finite Traces: Insensitivi...Knowledge Representation and Reasoning (KRR)AIW: AI for web services: semantic description...
113Programming by Example using Least General Gen...Applications (APP)\nHeuristic Search and Optim...APP: Intelligent User Interfaces\nAPP: Other A...
129Using Model-Based Diagnosis to Improve Softwar...Applications (APP)\nKnowledge Representation a...APP: Other Applications\nKRR: Automated Reason...
140Confident Reasoning on Raven’s Progressive Mat...Knowledge Representation and Reasoning (KRR)KRR: Geometric, Spatial, and Temporal Reasonin...
162SenticNet 3: A Common and Common-Sense Knowled...Cognitive Systems (CS)\nKnowledge Representati...CS: Conceptual inference and reasoning\nKRR: C...
168Backdoors to PlanningKnowledge Representation and Reasoning (KRR)\n...KRR: Computational Complexity of Reasoning\nPS...
170Datalog Rewritability of Disjunctive Datalog P...Knowledge Representation and Reasoning (KRR)KRR: Ontologies\nKRR: Automated Reasoning and ...
173The Most Uncreative Examinee: A First Step tow...Knowledge Representation and Reasoning (KRR)KRR: Automated Reasoning and Theorem Proving
174Acquiring Commonsense Knowledge for Sentiment ...Human-Computation and Crowd Sourcing (HCC)\nKn...HCC: Domain-specific implementation challenges...
179Explanation-Based Approximate Weighted Model C...Knowledge Representation and Reasoning (KRR)\n...KRR: Logic Programming\nRU: Probabilistic Infe...
180A Knowledge Compilation Map for Ordered Real-V...Knowledge Representation and Reasoning (KRR)KRR: Computational Complexity of Reasoning\nKR...
205A reasoner for the RCC-5 and RCC-8 calculi ext...Knowledge Representation and Reasoning (KRR)\n...KRR: Computational Complexity of Reasoning\nKR...
279Computing General First-order Parallel and Pri...Knowledge Representation and Reasoning (KRR)KRR: Common-Sense Reasoning\nKRR: Nonmonotonic...
287Data Quality in Ontology-based Data Access: Th...Knowledge Representation and Reasoning (KRR)APP: Other Applications\nKRR: Ontologies\nKRR:...
291Diagnosing Analogue Linear Systems Using Dynam...Knowledge Representation and Reasoning (KRR)KRR: Diagnosis and Abductive Reasoning
294Elementary Loops RevisitedKnowledge Representation and Reasoning (KRR)KRR: Logic Programming
296Joint Morphological Generation and Syntactic L...NLP and Knowledge Representation (NLPKR)NLPKR: Natural Language Processing (General/Ot...
308Implementing GOLOG in Answer Set ProgrammingKnowledge Representation and Reasoning (KRR)\n...KRR: Action, Change, and Causality\nKRR: Logic...
321Qualitative Reasoning with Modelica ModelsApplications (APP)\nKnowledge Representation a...APP: Other Applications\nKRR: Knowledge Repres...
324Pathway Specification and Comparative Queries:...Knowledge Representation and Reasoning (KRR)APP: Biomedical / Bioinformatics\nKRR: Knowled...
326Testable Implications of Linear Structural Equ...Knowledge Representation and Reasoning (KRR)\n...KRR: Action, Change, and Causality\nRU: Graphi...
348Exploiting Support Sets for Answer Set Program...Knowledge Representation and Reasoning (KRR)KRR: Ontologies\nKRR: Description Logics\nKRR:...
352Local-To-Global Consistency Implies Tractabili...Knowledge Representation and Reasoning (KRR)KRR: Computational Complexity of Reasoning\nKR...
356Exploring the Boundaries of Decidable Verifica...Knowledge Representation and Reasoning (KRR)KRR: Action, Change, and Causality\nKRR: Geome...
374Managing Change in Graph-structured Data Using...Knowledge Representation and Reasoning (KRR)KRR: Computational Complexity of Reasoning\nKR...
382Coactive Learning for Locally Optimal Problem ...Humans and AI (HAI)\nKnowledge Representation ...HCC: Active learning from imperfect human labe...
383Large Scale Analogical ReasoningCognitive Systems (CS)\nKnowledge Representati...CS: Conceptual inference and reasoning\nCS: St...
395Contraction and Revision over DL-Lite TBoxesKnowledge Representation and Reasoning (KRR)KRR: Belief Change\nKRR: Description Logics\nK...

通过查看每组聚类结果可知,每类结果有较为清晰的特征:

  • 0:该类主要包含 VIS 等视觉相关文章
  • 1:该类主要包含 AIW 及 ROB 等文章
  • 2:该类主要包含 NMLA 机器学习等文章
  • 3:该类主要包含 GTEP 游戏类文章
  • 4:该类主要包含 AIW 及社交网络等文章
  • 5:该类主要包含 SCS 和 HSO等搜索类文章
  • 6:该类主要包含 PS 及 CS 策略计划类文章
  • 7:该类主要包含 GTEP 等文章
  • 8:该类主要保护 APP 及 MLA等文章
  • 9:该类主要包含 KRR 知识表示与推理等文章
# 创建一个TSNE对象,指定要降维到的维数为2,随机数种子为RANDOM_STATE
tsne = sklearn.manifold.TSNE(n_components=2, random_state=RANDOM_STATE, init="random")# 调用TSNE对象的fit_transform方法,传入X_passage数据集,返回一个降维后的数据数组,赋值给X_tsne
X_tsne = tsne.fit_transform(X_passage)sns.scatterplot(x=X_tsne[:,0], y=X_tsne[:,1], hue=labels, palette="deep") # 散点图


请添加图片描述

从上图可知,通过作者、词干等分词后,聚类效果更好

5 聚类效果分析

本章分析不同k值对聚类效果的影响,以及该数据集中k取什么效果最好

k_range = range(5,15)
label_dict = {}
for k in k_range:model = KMeans(n_clusters=k,  init='k-means++', max_iter=100, n_init=1, random_state=RANDOM_STATE)model.fit(X_passage)labels = model.labels_label_dict[k]=labels
label_dict[7]
array([0, 0, 5, 6, 3, 3, 3, 0, 6, 3, 4, 0, 0, 6, 2, 3, 5, 6, 6, 3, 3, 1,5, 0, 0, 3, 6, 4, 6, 6, 5, 4, 0, 0, 5, 1, 0, 5, 6, 5, 0, 1, 0, 3,5, 1, 0, 3, 3, 4, 6, 6, 6, 3, 0, 6, 1, 5, 3, 0, 5, 0, 6, 3, 6, 5,1, 5, 3, 1, 5, 1, 0, 0, 6, 3, 1, 5, 3, 5, 6, 3, 6, 3, 3, 6, 0, 5,3, 0, 0, 6, 2, 3, 0, 4, 3, 5, 3, 3, 1, 6, 0, 6, 1, 4, 5, 2, 3, 1,5, 4, 3, 1, 5, 4, 6, 3, 0, 3, 3, 0, 3, 2, 5, 3, 0, 2, 0, 1, 0, 1,1, 0, 5, 5, 3, 3, 2, 3, 1, 0, 3, 6, 0, 3, 4, 0, 2, 5, 0, 3, 3, 5,0, 4, 6, 2, 0, 6, 0, 5, 1, 0, 4, 3, 5, 6, 2, 5, 1, 5, 2, 1, 1, 0,3, 4, 5, 1, 1, 5, 3, 3, 1, 4, 3, 6, 5, 5, 5, 5, 4, 0, 0, 1, 2, 5,4, 0, 4, 2, 3, 3, 6, 1, 2, 3, 3, 6, 0, 3, 1, 3, 3, 6, 2, 0, 2, 3,6, 3, 3, 3, 5, 0, 3, 6, 3, 3, 1, 3, 0, 5, 6, 4, 5, 2, 4, 3, 0, 3,0, 0, 6, 6, 3, 4, 0, 0, 4, 3, 5, 0, 0, 6, 2, 3, 0, 0, 0, 5, 6, 3,0, 5, 4, 0, 6, 6, 5, 0, 5, 4, 0, 0, 3, 4, 0, 1, 3, 2, 0, 4, 4, 3,0, 1, 0, 6, 6, 1, 4, 0, 1, 5, 3, 6, 0, 0, 0, 3, 0, 5, 5, 0, 6, 5,1, 3, 4, 1, 2, 5, 4, 2, 0, 5, 0, 4, 5, 1, 3, 1, 1, 1, 1, 0, 0, 0,5, 0, 3, 4, 4, 3, 5, 5, 4, 3, 1, 2, 5, 5, 3, 2, 2, 5, 1, 3, 3, 6,1, 0, 1, 0, 1, 2, 0, 5, 3, 3, 1, 3, 0, 0, 5, 6, 6, 3, 5, 4, 0, 3,1, 3, 6, 2, 3, 3, 2, 3, 1, 1, 3, 3, 4, 4, 0, 4, 5, 2, 0, 6, 6, 1,3, 0])
# 创建2行5列的子图布局
fig, axes = plt.subplots(2, 5, figsize=(25, 10))# 将10个子图填充到子图布局中
for k, label in label_dict.items():row, col = divmod(k-5, 5)  # 根据k计算在子图布局中的行和列位置ax = axes[row, col]sns.scatterplot(x=X_tsne[:, 0], y=X_tsne[:, 1], hue=label, palette="deep", ax=ax)ax.set_title("cluster = %d" % k)# 调整子图布局
plt.tight_layout()
plt.show()


请添加图片描述

# 创建一个TSNE对象,指定要降维到的维数为3,随机数种子为RANDOM_STATE
tsne = sklearn.manifold.TSNE(n_components=3, random_state=RANDOM_STATE, init="random")# 调用TSNE对象的fit_transform方法,传入X_passage数据集,返回一个降维后的数据数组,赋值给X_tsne
X_tsne = tsne.fit_transform(X_passage)# 创建一个大画布,包含10个子图
fig, axes = plt.subplots(2, 5, figsize=(25, 10), subplot_kw={'projection': '3d'})# 将10个子图填充到大画布中
for k, ax in zip(label_dict.keys(), axes.flatten()):# 绘制散点图,指定散点的大小ax.scatter(X_tsne[:, 0], X_tsne[:, 1], X_tsne[:, 2], c=label_dict[k], cmap='Dark2')# 设置标题,指定标题的字体大小ax.set_title("cluster = %d" % k, fontsize=16)# 调整子图布局
plt.tight_layout()
plt.show()


请添加图片描述

以上可见,用2d和3d图展示聚类效果,在5到14的Kmeans中没有聚类效果特别好的,但是感觉取7时聚类效果更好一点

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.mzph.cn/news/662940.shtml

如若内容造成侵权/违法违规/事实不符,请联系多彩编程网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

K8s 集群可观测性-数据分流最佳实践

简介 在微服务架构下&#xff0c;一个 k8s 集群中经常会部署多套业务&#xff0c;同时也意味着不同团队、不同角色、不同的业务会在同一集群中&#xff0c;需要将不同业务的数据在不同的空间进行管理和查看。 在传统的主机环境下&#xff0c;这个是可以通过不同的主机部署 Da…

《元梦之星》赛季更新带来“新”内容,为何却被玩家集体声讨?

前段时间&#xff0c;《元梦之星》迎来了“山海奇遇”赛季的重磅更新&#xff0c;诸多“新”内容的上线吸引了很多玩家们的关注&#xff0c;然而在新版本开启之后没有多&#xff0c;新玩法新时装甚至是游戏中的新改动都引起了不少玩家的不满。 在新赛季开启之后&#xff0c;玩家…

Python爬虫http基本原理

HTTP 基本原理 在本节中&#xff0c;我们会详细了解 HTTP 的基本原理&#xff0c;了解在浏览器中敲入 URL 到获取网页内容之间发生了什么。了解了这些内容&#xff0c;有助于我们进一步了解爬虫的基本原理。 2.1.1 URI 和 URL 这里我们先了解一下 URI 和 URL&#xff0c;URI…

抖音弹幕直播玩法汉字找不同文字找不同无人值执守自动玩游戏自带语音播报的开发日志

#找不同# 要解决如下几个问题&#xff1a; 1.声音sprite的录制和调用&#xff0c;解决方案以及解决库如下&#xff1a; howler.min.js://一款不错的音频播放js库。 2.鼠标自动飘浮,使用的库 anime.min.js 3.资源预加载 preload.min.js 4.其它使用到的库 jquery,vue

stack和queue及优先级队列和适配器(包括deque)的介绍

stack stack的介绍 stack是一种容器适配器&#xff0c;专门用在具有后进先出操作的上下文环境中&#xff0c;其删除只能从容器的一端进行元素的插入与提取操作。stack是作为容器适配器被实现的&#xff0c;容器适配器即是对特定类封装作为其底层的容器&#xff0c;并提供一组…

Android使用ScrollView导致鼠标点击事件无效

平台 测试平台: RK3288 Android8.1RK3588 Android 12 问题 首先, 这个问题的前提是, 使用的输入设备是**鼠标**, 普通的触摸屏并不会出现这个问题. 大致的流程是APP的UI布局中采用ScrollView作为根容器, 之后添加各类子控件, 在一起准备就绪后, 使用鼠标进行功能测试, 出现…

国产隔离芯片的质量控制与发展趋势

随着电子技术的飞速发展&#xff0c;国产隔离芯片在电力电子、通信设备等领域中扮演着重要角色。然而&#xff0c;随之而来的是对于其质量控制的迫切需求。本文将从结构、制造工艺、测试手段等方面对国产隔离芯片的质量控制进行分析&#xff0c;并展望其未来的发展趋势。 一、国…

element-ui link 组件源码分享

link 组件的 api 涉及的内容不是很多&#xff0c;源码部分的内容也相对较简单&#xff0c;下面从以下这三个方面来讲解&#xff1a; 一、组件结构 1.1 组件结构如下图&#xff1a; 二、组件属性 2.1 组件主要有 type、underline、disabled、href、icon 这些属性&#xff0c;…

KVM-安装-使用-迁移

一. KVM安装 1. 基础安装 # 下载源 curl -o /etc/yum.repos.d/Centos-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo# 安装基础软件 yum -y install tree vim wget bash-completion bash-completion-extras lrzsz net-tools sysstat iotop iftop htop unzip nc nmap …

批量修改文件后缀名

需要将/opt/module/test/路径下的txt文件后缀修改为cpp&#xff0c;并且以年份结尾 代码如下&#xff1a; #!/bin/bashyear2020 directory"/opt/module/test/"cd "$directory" || exit 1for name in *.txt; donew_name"${name%.txt}_${year}.cpp&qu…

SpringBoot security 安全认证(一)——登录验证

本节内容&#xff1a;使用springboot自动security模块实现用户登录验证功能&#xff1b; 登录过程如下图&#xff1a; AuthenticationManager内容实现用户账号密码验证&#xff0c;还可以对用户状态&#xff08;启用/禁用&#xff09;&#xff0c;逻辑删除&#xff0c;账号是否…

LeetCode.189. 轮转数组

题目 题目链接 分析 首先能想到的就是可以用一个新数组&#xff0c;先保存原数组的后 k 个元素&#xff0c;再保存原数组的前 n−k 个元素。但题目要求不使用额外的数组空间&#xff0c;那么就需要在原数组上做操作。 我们可以先把整个数组翻转一下&#xff0c;这样后半段元…

虚幻UE5Matehuman定制自己的虚拟人,从相机拍照到UE5制作全流程

开启自己的元宇宙,照片扫描真实的人类,生成虚拟形象,保姆级教程,欢迎大家指正。 需要的软件: 制作流程: 一.拍照。 围绕自己拍照,大概20多张图就差不多了,把脑门漏出来,无需拍后脑勺。 拍照方式 例如,拍照时尽量不要在脸上体现出明显的光源方向。

07. 【Linux教程】远程登录

Linux 远程登录 前面介绍了如何安装 Linux 终端工具&#xff0c;本小节介绍本地电脑如何使用 ssh 命令远程登录、Linux 终端工具远程登录的方式&#xff0c;这两种登录方式都是基于 ssh 网络安全协议的&#xff0c;学会使用远程登录 Linux 服务器&#xff0c;会让你对 Linux 系…

Postman(接口测试工具),什么是Postman接口

目录 一.基本介绍 Postman 是什么Postman 快速入门快速入门需求说明 二.Postman 完成 Controller 层测试 需要的代码&#xff1a; Java类request.jspsuccess.jsp1. 完成请求2. 完成请求3. 完成请求4. 完成请求5. 完成请求 三.发送join 目录 一.基本介绍 Postman 是什么 …

使用 Dockerfile 定制镜像详解

使用 Dockerfile 定制镜像详解 1.DockerfileFROM 指定基础镜像RUN 执行命令构建镜像 2.COPY 复制文件3.ADD 更高级的复制文件4.CMD 容器启动命令5.ENTRYPOINT 入口点6.ENV 设置环境变量7.ARG 构建参数8.VOLUME 定义匿名卷9.EXPOSE 暴露端口10.WORKDIR 指定工作目录11.USER 指定…

通过Netbackup恢复Oracle备份实操手册

1、系统环境描述 1 2、恢复前数据备份 2 2.1 在NBU上执行一次完整的备份 2 2.2 查看ORACLE的备份集 3 2.2.1在备份客户端上查看备份集 3 2.2.2在备份服务器netbackup上查看客户端备份集 4 3、本机恢复方法 5 3.1丢失SPFILE文件恢复方法 5 3.2丢失CONTROLFILE文件恢复方…

51单片机编程应用(C语言):模块化编程

下面我们模块化几个函数&#xff1a; Delay.c //延时子函数 void Delay(unsigned int xms) {unsigned char i, j;while(xms--){i 2;j 239;do{while (--j);} while (--i);} } Delay.h #ifndef __DELAY_H__ #define __DELAY_H__void Delay(unsigned int xms);#endifNixie.h …

js 设置、获取、删除标签属性以及H5自定义属性

1. 设置标签属性 使用setAttribute()(‘属性名’, ‘属性值’)方法可以添加、修改、删除属性。   下面的demo是为input添加、修改、删除value属性&#xff1a; 1.1. HTML <input type"text" class"input"> <input type"text" class…

有没有合适写毕业论文的AI工具?

最近挺多同学在忙着写毕业论文&#xff0c;不断在“提交-打回-修改-提交”过程里循环着&#xff0c;好不容易写完了&#xff0c;还得考虑论文查重的问题&#xff01;基哥作为一名曾经的毕业生&#xff0c;当然也体验过这种痛苦了。 但是&#xff0c;大人&#xff0c;时代变了&…