首页 > pyspark > PySpark:时间戳转换的字符串

PySpark:时间戳转换的字符串 (PySpark: String to timestamp transformation)

2019-03-07 pyspark

问题

我正在处理时间数据并尝试将字符串转换为时间戳格式。

以下是“时间”列的外观

+----------+
|  Time    |
+----------+
|1358380800|
|1380672000|
+----------+

这就是我想要的

+---------------+
|  Time         |
+---------------+
|2013/1/17 8:0:0|
|2013/10/2 8:0:0|
+---------------+

我找到了一些类似的问题和答案,并尝试了这些代码,但都以'null'结尾

df2 = df.withColumn("Time", test["Time"].cast(TimestampType()))

df2 = df.withColumn('Time', F.unix_timestamp('Time', 'yyyy-MM-dd').cast(TimestampType()))

解决方法

好吧,你正在做相反的事情。sql函数unix_timestamp将具有给定格式的字符串转换为unix时间戳。如果要将unix时间戳转换为datetime格式,则必须使用from_unixtime sql函数:

from pyspark.sql import functions as F
from pyspark.sql import types as T
l1 = [('1358380800',),('1380672000',)]
df = spark.createDataFrame(l1,['Time'])
df.withColumn('Time', F.from_unixtime(df.Time).cast(T.TimestampType())).show()

输出:

+-------------------+ 
|               Time|
+-------------------+
|2013-01-17 01:00:00|
|2013-10-02 02:00:00|
+-------------------+

问题

I am working with time data and try to convert the string to timestamp format.

Here is what the 'Time' column looks like

+----------+
|  Time    |
+----------+
|1358380800|
|1380672000|
+----------+

Here is what I want

+---------------+
|  Time         |
+---------------+
|2013/1/17 8:0:0|
|2013/10/2 8:0:0|
+---------------+

I find some similar questions and answers and have tried these code, but all end with 'null'

df2 = df.withColumn("Time", test["Time"].cast(TimestampType()))

df2 = df.withColumn('Time', F.unix_timestamp('Time', 'yyyy-MM-dd').cast(TimestampType()))

解决方法

Well your are doing it the other way around. The sql function unix_timestamp converts a string with the given format to a unix timestamp. When you want to convert a unix timestamp to the datetime format, you have to use the from_unixtime sql function:

from pyspark.sql import functions as F
from pyspark.sql import types as T
l1 = [('1358380800',),('1380672000',)]
df = spark.createDataFrame(l1,['Time'])
df.withColumn('Time', F.from_unixtime(df.Time).cast(T.TimestampType())).show()

Output:

+-------------------+ 
|               Time|
+-------------------+
|2013-01-17 01:00:00|
|2013-10-02 02:00:00|
+-------------------+
相似信息