Code.djangoproject.com



Error in Running the pyspark in Django application In DjangoSolution :read_sparks.pyfrom pyspark.sql import SparkSessionspark = SparkSession.builder.appName("Yaanal").master("local[*]").getOrCreate()def read_csv(fb):return spark.read.csv(fb)views.pyfrom read_sparks import *………def someviews(request):‘’’’get the file name dynamically’’’fb=”C:\Users\Dell\myenv\Lib\dist-packages\yaanal\media\user_2\all_datas.csv”df=read_csv(fb)………..…..settings.py………..if 'SPARK_HOME' not in os.environ:os.environ['SPARK_HOME'] = 'C:\spark\spark'SPARK_HOME = os.environ['SPARK_HOME'] HADOOP_HOME = os.environ['HADOOP_HOME']SCALA_HOME = os.environ['SCALA_HOME']JAVA_HOME = os.environ['JAVA_HOME']PY4J_DIR = os.path.normpath(SPARK_HOME+'/python/build')PYSPARK_DIR = os.path.normpath(SPARK_HOME+'/python')sys.path.insert(0,PY4J_DIR)sys.path.insert(0,PYSPARK_DIR)………….Output:In Normal ProgramSolution :read_sparks.pyfrom pyspark.sql import SparkSessionspark = SparkSession.builder.appName("Yaanal").master("local[*]").getOrCreate()def read_csv(fb):return spark.read.csv(fb)fb=”C:\Users\Dell\myenv\Lib\dist-packages\yaanal\media\user_2\all_datas.csv”print read_csv(fb)Output:The problem here is the program runs in local gives correct expected output and the same program runs in Django application gives error. Please help me to fix the problem. ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download