pyspark使用 graphframes创建图的方法

1、安装graphframes的步骤

1.1 查看 spark 和 scala版本

在终端输入： spark-shell --version 查看spark 和scala版本

1.2 在maven库中下载对应版本的graphframes

https://mvnrepository.com/artifact/graphframes/graphframes

我这里需要的是spark 2.4 scala 2.11版本

https://mvnrepository.com/artifact/graphframes/graphframes/0.8.0-spark2.4-s_2.11

1.3 在pyspark的环境中配置graphframe的jar包

os.environ['PYSPARK_PYTHON'] = 'Python3.7/bin/python'
os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars graphframes-0.8.1-spark2.4-s_2.11.jar pyspark-shell'spark = SparkSession \.builder \.appName("read_data") \.config('spark.pyspark.python', 'Python3.7/bin/python') \.config('spark.yarn.dist.archives', 'hdfs://ns62007/user/dmc_adm/_PYSPARK_ENV/Python3.7.zip#Python3.7') \.config('spark.executorEnv.PYSPARK_PYTHON', 'Python3.7/bin/python') \.config('spark.sql.autoBroadcastJoinThreshold', '-1') \.enableHiveSupport() \.getOrCreate()spark.sparkContext.addPyFile('graphframes-0.8.1-spark2.4-s_2.11.jar')

2、导入GraphFrame创建图

2.1 导入包使用

from graphframes import GraphFrame

2.2 创建图的例子

from pyspark.sql.types import *
import pandas as pd
from graphframes import GraphFrame#创建图的方法1
v = spark.createDataFrame([("a", "Alice", 34),("b", "Bob", 36),("c", "Charlie", 30),
], ["id", "name", "age"])# Create an Edge DataFrame with "src" and "dst" columns
e = spark.createDataFrame([("a", "b", "friend"),("b", "c", "follow"),("c", "b", "follow"),
], ["src", "dst", "relationship"])
# Create a GraphFrame
g = GraphFrame(v, e)# Query: Get in-degree of each vertex.
g.inDegrees.show()

也可以简单化顶点和边：

#创建图的方法2
edges_df= spark.createDataFrame([("a", "b"),("b", "c"),("c", "b"),
], ["src", "dst"])
nodes_df=spark.createDataFrame([(1, "a"),(2, "b"),(3, "c")
], ["num","id"])graph=GraphFrame(nodes_df, edges_df)
graph.inDegrees.show()