Spark SQL----内置函数Csv Functions Misc Functions

Spark SQL----内置函数Csv Functions& Misc Functions

  • Csv Functions
    例子:
-- from_csv
SELECT from_csv('1, 0.8', 'a INT, b DOUBLE');
+----------------+
|from_csv(1, 0.8)|
+----------------+
|        {1, 0.8}|
+----------------+SELECT from_csv('26/08/2015', 'time Timestamp', map('timestampFormat', 'dd/MM/yyyy'));
+--------------------+
|from_csv(26/08/2015)|
+--------------------+
|{2015-08-26 00:00...|
+--------------------+-- schema_of_csv
SELECT schema_of_csv('1,abc');
+--------------------+
|schema_of_csv(1,abc)|
+--------------------+
|STRUCT<_c0: INT, ...|
+--------------------+-- to_csv
SELECT to_csv(named_struct('a', 1, 'b', 2));
+--------------------------------+
|to_csv(named_struct(a, 1, b, 2))|
+--------------------------------+
|                             1,2|
+--------------------------------+SELECT to_csv(named_struct('time', to_timestamp('2015-08-26', 'yyyy-MM-dd')), map('timestampFormat', 'dd/MM/yyyy'));
+----------------------------------------------------------------+
|to_csv(named_struct(time, to_timestamp(2015-08-26, yyyy-MM-dd)))|
+----------------------------------------------------------------+
|                                                      26/08/2015|
+----------------------------------------------------------------+
  • Misc Functions
    例子:
-- aes_decrypt
SELECT aes_decrypt(unhex('83F16B2AA704794132802D248E6BFD4E380078182D1544813898AC97E709B28A94'), '0000111122223333');
+------------------------------------------------------------------------------------------------------------------------+
|aes_decrypt(unhex(83F16B2AA704794132802D248E6BFD4E380078182D1544813898AC97E709B28A94), 0000111122223333, GCM, DEFAULT, )|
+------------------------------------------------------------------------------------------------------------------------+
|                                                                                                        [53 70 61 72 6B]|
+------------------------------------------------------------------------------------------------------------------------+SELECT aes_decrypt(unhex('6E7CA17BBB468D3084B5744BCA729FB7B2B7BCB8E4472847D02670489D95FA97DBBA7D3210'), '0000111122223333', 'GCM');
+--------------------------------------------------------------------------------------------------------------------------------+
|aes_decrypt(unhex(6E7CA17BBB468D3084B5744BCA729FB7B2B7BCB8E4472847D02670489D95FA97DBBA7D3210), 0000111122223333, GCM, DEFAULT, )|
+--------------------------------------------------------------------------------------------------------------------------------+
|                                                                                                            [53 70 61 72 6B 2...|
+--------------------------------------------------------------------------------------------------------------------------------+SELECT aes_decrypt(unbase64('3lmwu+Mw0H3fi5NDvcu9lg=='), '1234567890abcdef', 'ECB', 'PKCS');
+------------------------------------------------------------------------------+
|aes_decrypt(unbase64(3lmwu+Mw0H3fi5NDvcu9lg==), 1234567890abcdef, ECB, PKCS, )|
+------------------------------------------------------------------------------+
|                                                          [53 70 61 72 6B 2...|
+------------------------------------------------------------------------------+SELECT aes_decrypt(unbase64('2NYmDCjgXTbbxGA3/SnJEfFC/JQ7olk2VQWReIAAFKo='), '1234567890abcdef', 'CBC');
+-----------------------------------------------------------------------------------------------------+
|aes_decrypt(unbase64(2NYmDCjgXTbbxGA3/SnJEfFC/JQ7olk2VQWReIAAFKo=), 1234567890abcdef, CBC, DEFAULT, )|
+-----------------------------------------------------------------------------------------------------+
|                                                                                 [41 70 61 63 68 6...|
+-----------------------------------------------------------------------------------------------------+SELECT aes_decrypt(unbase64('AAAAAAAAAAAAAAAAAAAAAPSd4mWyMZ5mhvjiAPQJnfg='), 'abcdefghijklmnop12345678ABCDEFGH', 'CBC', 'DEFAULT');
+---------------------------------------------------------------------------------------------------------------------+
|aes_decrypt(unbase64(AAAAAAAAAAAAAAAAAAAAAPSd4mWyMZ5mhvjiAPQJnfg=), abcdefghijklmnop12345678ABCDEFGH, CBC, DEFAULT, )|
+---------------------------------------------------------------------------------------------------------------------+
|                                                                                                     [53 70 61 72 6B]|
+---------------------------------------------------------------------------------------------------------------------+SELECT aes_decrypt(unbase64('AAAAAAAAAAAAAAAAQiYi+sTLm7KD9UcZ2nlRdYDe/PX4'), 'abcdefghijklmnop12345678ABCDEFGH', 'GCM', 'DEFAULT', 'This is an AAD mixed into the input');
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
|aes_decrypt(unbase64(AAAAAAAAAAAAAAAAQiYi+sTLm7KD9UcZ2nlRdYDe/PX4), abcdefghijklmnop12345678ABCDEFGH, GCM, DEFAULT, This is an AAD mixed into the input)|
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                                                                        [53 70 61 72 6B]|
+--------------------------------------------------------------------------------------------------------------------------------------------------------+-- aes_encrypt
SELECT hex(aes_encrypt('Spark', '0000111122223333'));
+-----------------------------------------------------------+
|hex(aes_encrypt(Spark, 0000111122223333, GCM, DEFAULT, , ))|
+-----------------------------------------------------------+
|                                       F30133168D792EAF3...|
+-----------------------------------------------------------+SELECT hex(aes_encrypt('Spark SQL', '0000111122223333', 'GCM'));
+---------------------------------------------------------------+
|hex(aes_encrypt(Spark SQL, 0000111122223333, GCM, DEFAULT, , ))|
+---------------------------------------------------------------+
|                                           71E5323818BDA5893...|
+---------------------------------------------------------------+SELECT base64(aes_encrypt('Spark SQL', '1234567890abcdef', 'ECB', 'PKCS'));
+---------------------------------------------------------------+
|base64(aes_encrypt(Spark SQL, 1234567890abcdef, ECB, PKCS, , ))|
+---------------------------------------------------------------+
|                                           3lmwu+Mw0H3fi5NDv...|
+---------------------------------------------------------------+SELECT base64(aes_encrypt('Apache Spark', '1234567890abcdef', 'CBC', 'DEFAULT'));
+---------------------------------------------------------------------+
|base64(aes_encrypt(Apache Spark, 1234567890abcdef, CBC, DEFAULT, , ))|
+---------------------------------------------------------------------+
|                                                 XPholDIX2Fq0MIEf4...|
+---------------------------------------------------------------------+SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH', 'CBC', 'DEFAULT', unhex('00000000000000000000000000000000')));
+---------------------------------------------------------------------------------------------------------------------+
|base64(aes_encrypt(Spark, abcdefghijklmnop12345678ABCDEFGH, CBC, DEFAULT, unhex(00000000000000000000000000000000), ))|
+---------------------------------------------------------------------------------------------------------------------+
|                                                                                                 AAAAAAAAAAAAAAAAA...|
+---------------------------------------------------------------------------------------------------------------------+SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH', 'GCM', 'DEFAULT', unhex('000000000000000000000000'), 'This is an AAD mixed into the input'));
+------------------------------------------------------------------------------------------------------------------------------------------------+
|base64(aes_encrypt(Spark, abcdefghijklmnop12345678ABCDEFGH, GCM, DEFAULT, unhex(000000000000000000000000), This is an AAD mixed into the input))|
+------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                                                            AAAAAAAAAAAAAAAAQ...|
+------------------------------------------------------------------------------------------------------------------------------------------------+-- assert_true
SELECT assert_true(0 < 1);
+--------------------------------------------+
|assert_true((0 < 1), '(0 < 1)' is not true!)|
+--------------------------------------------+
|                                        NULL|
+--------------------------------------------+-- bitmap_bit_position
SELECT bitmap_bit_position(1);
+----------------------+
|bitmap_bit_position(1)|
+----------------------+
|                     0|
+----------------------+SELECT bitmap_bit_position(123);
+------------------------+
|bitmap_bit_position(123)|
+------------------------+
|                     122|
+------------------------+-- bitmap_bucket_number
SELECT bitmap_bucket_number(123);
+-------------------------+
|bitmap_bucket_number(123)|
+-------------------------+
|                        1|
+-------------------------+SELECT bitmap_bucket_number(0);
+-----------------------+
|bitmap_bucket_number(0)|
+-----------------------+
|                      0|
+-----------------------+-- bitmap_count
SELECT bitmap_count(X '1010');
+---------------------+
|bitmap_count(X'1010')|
+---------------------+
|                    2|
+---------------------+SELECT bitmap_count(X 'FFFF');
+---------------------+
|bitmap_count(X'FFFF')|
+---------------------+
|                   16|
+---------------------+SELECT bitmap_count(X '0');
+-------------------+
|bitmap_count(X'00')|
+-------------------+
|                  0|
+-------------------+-- current_catalog
SELECT current_catalog();
+-----------------+
|current_catalog()|
+-----------------+
|    spark_catalog|
+-----------------+-- current_database
SELECT current_database();
+------------------+
|current_database()|
+------------------+
|           default|
+------------------+-- current_schema
SELECT current_schema();
+------------------+
|current_database()|
+------------------+
|           default|
+------------------+-- current_user
SELECT current_user();
+--------------+
|current_user()|
+--------------+
|  jungtaek.lim|
+--------------+-- equal_null
SELECT equal_null(3, 3);
+----------------+
|equal_null(3, 3)|
+----------------+
|            true|
+----------------+SELECT equal_null(1, '11');
+-----------------+
|equal_null(1, 11)|
+-----------------+
|            false|
+-----------------+SELECT equal_null(true, NULL);
+----------------------+
|equal_null(true, NULL)|
+----------------------+
|                 false|
+----------------------+SELECT equal_null(NULL, 'abc');
+---------------------+
|equal_null(NULL, abc)|
+---------------------+
|                false|
+---------------------+SELECT equal_null(NULL, NULL);
+----------------------+
|equal_null(NULL, NULL)|
+----------------------+
|                  true|
+----------------------+-- hll_sketch_estimate
SELECT hll_sketch_estimate(hll_sketch_agg(col)) FROM VALUES (1), (1), (2), (2), (3) tab(col);
+--------------------------------------------+
|hll_sketch_estimate(hll_sketch_agg(col, 12))|
+--------------------------------------------+
|                                           3|
+--------------------------------------------+-- hll_union
SELECT hll_sketch_estimate(hll_union(hll_sketch_agg(col1), hll_sketch_agg(col2))) FROM VALUES (1, 4), (1, 4), (2, 5), (2, 5), (3, 6) tab(col1, col2);
+-----------------------------------------------------------------------------------------+
|hll_sketch_estimate(hll_union(hll_sketch_agg(col1, 12), hll_sketch_agg(col2, 12), false))|
+-----------------------------------------------------------------------------------------+
|                                                                                        6|
+-----------------------------------------------------------------------------------------+-- input_file_block_length
SELECT input_file_block_length();
+-------------------------+
|input_file_block_length()|
+-------------------------+
|                       -1|
+-------------------------+-- input_file_block_start
SELECT input_file_block_start();
+------------------------+
|input_file_block_start()|
+------------------------+
|                      -1|
+------------------------+-- input_file_name
SELECT input_file_name();
+-----------------+
|input_file_name()|
+-----------------+
|                 |
+-----------------+-- java_method
SELECT java_method('java.util.UUID', 'randomUUID');
+---------------------------------------+
|java_method(java.util.UUID, randomUUID)|
+---------------------------------------+
|                   fe37ceed-2d82-464...|
+---------------------------------------+SELECT java_method('java.util.UUID', 'fromString', 'a5cf6c42-0c85-418f-af6c-3e4e5b1328f2');
+-----------------------------------------------------------------------------+
|java_method(java.util.UUID, fromString, a5cf6c42-0c85-418f-af6c-3e4e5b1328f2)|
+-----------------------------------------------------------------------------+
|                                                         a5cf6c42-0c85-418...|
+-----------------------------------------------------------------------------+-- monotonically_increasing_id
SELECT monotonically_increasing_id();
+-----------------------------+
|monotonically_increasing_id()|
+-----------------------------+
|                            0|
+-----------------------------+-- reflect
SELECT reflect('java.util.UUID', 'randomUUID');
+-----------------------------------+
|reflect(java.util.UUID, randomUUID)|
+-----------------------------------+
|               7fec8bae-b80b-448...|
+-----------------------------------+SELECT reflect('java.util.UUID', 'fromString', 'a5cf6c42-0c85-418f-af6c-3e4e5b1328f2');
+-------------------------------------------------------------------------+
|reflect(java.util.UUID, fromString, a5cf6c42-0c85-418f-af6c-3e4e5b1328f2)|
+-------------------------------------------------------------------------+
|                                                     a5cf6c42-0c85-418...|
+-------------------------------------------------------------------------+-- spark_partition_id
SELECT spark_partition_id();
+--------------------+
|SPARK_PARTITION_ID()|
+--------------------+
|                   0|
+--------------------+-- try_aes_decrypt
SELECT try_aes_decrypt(unhex('6E7CA17BBB468D3084B5744BCA729FB7B2B7BCB8E4472847D02670489D95FA97DBBA7D3210'), '0000111122223333', 'GCM');
+------------------------------------------------------------------------------------------------------------------------------------+
|try_aes_decrypt(unhex(6E7CA17BBB468D3084B5744BCA729FB7B2B7BCB8E4472847D02670489D95FA97DBBA7D3210), 0000111122223333, GCM, DEFAULT, )|
+------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                                                [53 70 61 72 6B 2...|
+------------------------------------------------------------------------------------------------------------------------------------+SELECT try_aes_decrypt(unhex('----------468D3084B5744BCA729FB7B2B7BCB8E4472847D02670489D95FA97DBBA7D3210'), '0000111122223333', 'GCM');
+------------------------------------------------------------------------------------------------------------------------------------+
|try_aes_decrypt(unhex(----------468D3084B5744BCA729FB7B2B7BCB8E4472847D02670489D95FA97DBBA7D3210), 0000111122223333, GCM, DEFAULT, )|
+------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                                                                NULL|
+------------------------------------------------------------------------------------------------------------------------------------+-- typeof
SELECT typeof(1);
+---------+
|typeof(1)|
+---------+
|      int|
+---------+SELECT typeof(array(1));
+----------------+
|typeof(array(1))|
+----------------+
|      array<int>|
+----------------+-- user
SELECT user();
+--------------+
|current_user()|
+--------------+
|  jungtaek.lim|
+--------------+-- uuid
SELECT uuid();
+--------------------+
|              uuid()|
+--------------------+
|55eab4e0-7a6a-444...|
+--------------------+-- version
SELECT version();
+--------------------+
|           version()|
+--------------------+
|3.5.1 fd86f85e181...|
+--------------------+

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.mzph.cn/bicheng/42311.shtml

如若内容造成侵权/违法违规/事实不符,请联系多彩编程网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

《妃梦千年》第二十六章:风暴前夕

第二十六章&#xff1a;风暴前夕 经过一系列的斗争&#xff0c;宫中的局势暂时得到了稳定。然而&#xff0c;林清婉知道&#xff0c;这一切只是表面的平静&#xff0c;敌人不会轻易放弃&#xff0c;他们必定会再次发动进攻。为了迎接即将到来的挑战&#xff0c;林清婉决定进一…

基于CentOS Stream 9平台搭建RabbitMQ3.13.4以及开机自启

1. erlang与RabbitMQ对应版本参考&#xff1a;https://www.rabbitmq.com/which-erlang.html 2. 安装erlang 官网&#xff1a;https://www.erlang.org/downloads GitHub: https://github.com/rabbitmq/erlang-rpm/releases 2.1 安装依赖&#xff1a; yum -y install gcc glib…

Nginx实战:nginx性能压测(ab)

在nginx的生产实践中,不管是服务上线,还是性能优化,都会遇到需要对nginx的性能压测,本文介绍一个简单的压测工具:ab命令 ab(Apache Bench)是一个常用的HTTP压力测试工具,可以用来测试Nginx的性能和压力。ab命令可以指定并发请求数、请求数、请求类型等参数,并输出测试…

显示近期用户登录情况的命令last

显示近期用户登录情况的命令last There is no nutrition in the blog content. After reading it, you will not only suffer from malnutrition, but also impotence. The blog content is all parallel goods. Those who are worried about being cheated should leave quick…

Spring Boot中集成DJL运行Python PyTorch模型:MNIST实战

Java Spring Boot 使用DJL 部署python训练的PyTorch模型&#xff08;MNIST&#xff09; Python 训练Pytorch模型Java Spring Boot使用DJL调用模型 pom.xml TranslatorCriteriaNDArray 创建NDArray(Tensor) 变更数据类型运算切片赋值翻转 PredictorMNIST实战 环境 Java 使用 …

算法力扣刷题总结篇 ——【四】

前言 栈和队列篇题目学习结束。 总结&#xff1a; 一、结构基础 stack类 &#xff08;1&#xff09;使用栈&#xff08;后入先出&#xff09;的结构时&#xff0c;用它。 &#xff08;2&#xff09;容器适配器&#xff08;container adaptor&#xff09;。本身不算容器&…

550kg级大载重长航时无人机直升机技术详解

550kg级大载重长航时无人机直升机&#xff0c;作为一种高性能的无人机系统&#xff0c;具备了多项先进的技术特点&#xff0c;以满足高海拔、高寒等复杂环境下的应用需求。这些无人机直升机通常具备高载重、长航时、强适应性、高可靠性和良好的任务拓展性。 设备由无人直升机平…

【面试八股文】java基础知识

引言 本文是java面试时的一些常见知识点总结归纳和一些拓展&#xff0c;笔者在学习这些内容时&#xff0c;特地整理记录下来&#xff0c;以供大家学习共勉。 一、数据类型 1.1 为什么要设计封装类&#xff0c;Integer和int区别是什么&#xff1f; 使用封装类的目的 对象化:…

数据结构——单链表专题

前言 本篇我们继续来介绍数据结构的知识——链表&#xff0c;这与顺序表是一个类别的知识&#xff0c;同样也非常重要&#xff0c;所以大家需要掌握好链表的内容&#xff0c;这对我们后面数据结构的学习也很重要&#xff0c;如果你对本篇的内容感兴趣&#xff0c;希望大佬可以…

使用RabbitMQ实现可靠的消息传递机制

使用RabbitMQ实现可靠的消息传递机制 大家好&#xff0c;我是微赚淘客系统3.0的小编&#xff0c;也是冬天不穿秋裤&#xff0c;天冷也要风度的程序猿&#xff01; 1. RabbitMQ简介 RabbitMQ是一个开源的消息代理软件&#xff0c;实现了高级消息队列协议&#xff08;AMQP&…

四大常见的排序算法JAVA

1. 冒泡排序 相邻的元素两两比较&#xff0c;大的放右边&#xff0c;小的放左边 第一轮比较完毕之后&#xff0c;最大值就已经确定&#xff0c;第二轮可以少循环一次&#xff0c;后面以此类推 如果数组中有n个数据&#xff0c;总共我们只要执行n-1轮的代码就可以 package Bu…

mq虚拟主机实现数据隔离

如果一个mq系统需要被多个实际业务系统共用&#xff0c;那么我们需要对于不同的业务系统分设不同的虚拟机进行数据隔离 本文将主要演示如何使用mq的web端进行创建不同虚拟机完成数据隔离 首先我们需要针对新的业务系统创建一个专属的用户&#xff0c;并赋予admin权限&#xf…

【matlab】状态空间模型与传递函数模型的建立与转换

目录 SISO系统 MIMO系统 状态空间模型 状态空间模型到传递函数模型的转换 传递函数模型到状态空间模型的转换 (1) 转换函数ss() (2) 规范形转换函数canon() (3) 常微分方程(传递函数)转换为状态空间模型函数dif2ss() 状态空间模型的变换 特征值、特征向量与广义特征向量的计算…

everything高级搜索-cnblog

everything高级搜索用法 基础4选项验证 总结搜索方式 高级搜索搜指定路径文件名: 文件名 路径不含文件名: &#xff01;文件名包含单词 路径包含指定内容: 路径 content:内容 大小写 区分大小写搜索搜指定路径文件名: case:文件名 路径全字匹配 全字搜指定路径文件名: wholewo…

力扣刷题--908. 最小差值 I【简单】

题目描述 给你一个整数数组 nums&#xff0c;和一个整数 k 。 在一个操作中&#xff0c;您可以选择 0 < i < nums.length 的任何索引 i 。将 nums[i] 改为 nums[i] x &#xff0c;其中 x 是一个范围为 [-k, k] 的整数。对于每个索引 i &#xff0c;最多 只能 应用 一次…

中霖教育:二级建造师未注册还需要继续教育吗?

关键词&#xff1a;中霖教育怎么样&#xff0c;中霖教育口碑 如果通过了二级建造师考试但是没有注册&#xff0c;还用继续教育吗? 1. 未注册的二级建造师 二级建造师在其证书获取后三年内没有进行注册时&#xff0c;在申请初始注册之前必须完成规定的本专业继续教育课程。 …

ubuntu运行qq音乐闪退

ubuntu运行qq音乐闪退 修改/usr/share/applications中的qqmusic.desktop&#xff0c;在Exec后加上 --no-sandbox,如下图所示&#xff1a; 该文件有可能是只读&#xff0c;权限不够的话用sudo vim qqmusic.desktop

java项目的一些功能(完善登录功能、注册接口参数校验、完善分页查询、完善日期格式、更新文章分类和添加文章分类的分组校验、自定义校验、文件上传 )

目录 完善登录功能 注册接口参数校验 完善分页查询 完善日期格式 更新文章分类和添加文章分类的分组校验 ​编辑 自定义校验 文件上传 完善登录功能 对前端传过来的明文密码进行md5加密处理 password DigestUtils.md5DigestAsHex(password.getBytes()); 这样既可 注…

Python数据分析案例50——基于EEMD-LSTM的石油价格预测

案例背景 很久没更新时间序列预测有关的东西了。 之前写了很多CNN-LSTM&#xff0c;GRU-attention&#xff0c;这种神经网络之内的不同模型的缝合&#xff0c;现在写一个模态分解算法和神经网络的缝合。 虽然eemd-lstm已经在学术界被做烂了&#xff0c;但是还是很多新手小白或…

驾校管理系统的全面革新与升级

智慧驾校系统是一款专为现代驾校量身定制的综合性管理平台,它深度融合了云计算、大数据、物联网及人工智能等前沿技术,旨在为驾校打造一个高效、智能、便捷的运营生态系统。该系统通过数字化、信息化的手段,彻底革新了传统驾校的管理模式,不仅极大地提升了驾校的运营效率,…