spot7.org logo
Home PHP C# C++ Android Java Javascript Python IOS SQL HTML Categories

Apache Spark map and reduce with passing values


Maybe you could use a class and work with it through the flow. I mean, define RevenueHour class

case class RevenueHour(date:
java.util.Date,revenue: Long, id: String)

Then built an intermediate RevenueHour in the map phase and then another one in the reduce phase.

val map: RDD[(Date, RevenueHour)] =
orders.map(row => 
  (
    getDateAs("hour", oo.getDate("date")), 
    RevenueHour(
      row.getDate("date"),
     
List(row.getIntOption("f2"),row.getIntOption("f3"),row.getIntOption("f4")).flatten.reduce(_
+ _),
      row.getString("id")
    )
  )
).reduceByKey((o1: RevenueHour, o2: RevenueHour)
=> RevenueHour(getDateAs("hour", o1.date),
o1.revenue + o2.revenue, o1.id))

I use o1 RevenueHour because both o1 and o2 will have same key and same id (because the where clause before).

Hope it helps.


Categories : Scala

Related to : Apache Spark map and reduce with passing values
Reduce a key-value pair into a key-list pair with Apache Spark
Map and ReduceByKey Input type and output type of reduce must be the same, therefore if you want to aggregate a list, you have to map the input to lists. Afterwards you combine the lists into one list. Combining lists You'll need a method to combine lists into one list. Phyton provides some methods to combine lists. append modifies the first list and will always return None. x = [1, 2, 3] x.a

Categories : Python
Apache spark - dealing with auto-updating inputs
Spark alone cannot recognize if a file has been updated. It does its job when reading for a first time the file and that's all. By default, Spark won't know that a file has been updated and won't know which parts of the file are updates. You should rather work with folders, Spark can run on a folder and can recognize if there is a new file to process in it -> sc.textFile(PATH_FOLDER)...

Categories : Hadoop
Passing values within JSP using Spring-MVC
The error was thrown because the controller expects the url as @RequestMapping(value = "/newpassword/{id}") and you are sending it as localhost:8085/newpassword/usernameAndDateToken. If you are trying to pass some values to the controller can pass it with the url and access it using @RequestParam annotation. simply as, localhost:8085/newpassword/53587359878397 and in controller, @RequestMappi

Categories : Jsp
Passing values to input through g:each iteration
Assuming your model contains the following def myAction() { [value: ['TEST1', 'TEST2', 'TEST3']] } This should do it <g:each status="i" in="${value}" var="item"> <g:textField id="justATest${i}" name='justATest' value="${item}"> </g:each>

Categories : Grails
Call a Package from Apex passing two values
To receive 2 values in package you obviously need procedure with 2 parameters. For example for your package: create or replace PACKAGE "PKG_MAIL_SEND" as -- let this procedure have 2 parameters: PROCEDURE proc_mail_send(p_address in varchar2, p_topic in varchar2); PROCEDURE proc_job_mail_send; PROCEDURE proc_kill_job_mail; end PKG__MAIL_SEND; First way After that click on Create page

Categories : Oracle
Recently Add
head :: tail pattern matching for strings
What is the easiest and most efficient way to make a min heap in Scala?
Removing mutable collections and making it look more scala style
Reverse list of n elements
scalac: Error: object CharRef in intelliJ 14
Scala - trouble with type inference in lambda expression
Exception on spark test
How can I emulate Haskell's typeclasses in Scala?
Slick: CRUD extension: How to encapsulate implicit mapping:BaseColumnType[T]
Can't find Traverse for sequencing Seq[ValidationNel[String, MyCaseClass]] => ValidationNel[String, Seq[MyCaseClass]]
Tail recursion: internal "loop" function or default values for accumulators
Scala - Add element:MyType to Array = option[MyType] expected
Scala, Composing Function with two values
Using dependent type to generate compile error
How to match all words in a sentence with scala combinators?
Parser Alternative Operator | Fails
ScalaTest assertion mismatch due to Physical Address
Scala implicit parameter and japanese smiley 'foldLeft'
Is it possible to user reduceByKey((x, y, z) => ...)?
How to implement security Authorization using scala and play?
SSO login using scala script
Sum elements based on a predicate
Keep track of completed Futures
API Observable with dynamic caching
java.io.IOException: Remotely closed in gatling
Scala permutations using two lists
Is Scala Either really a Monad
Spark: Use of distinct
Identifying two type wildcards as identical
how to package spark scala application
© Copyright 2017 spot7.org Publishing Limited. All rights reserved.