Berbagi teknologi

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

Di spark sql, pengguna dapat menggunakan petunjuk Gabung untuk menyarankan jenis Gabung Spark mana yang harus digunakan. Sebelum Spark 3.0, hanya petunjuk Gabung seperti BROADCAST yang didukung. Mulai dari Spark 3.0, tiga Petunjuk Gabung, MERGE, SHUFFLE_HASH dan SHUFFLE_REPLICATE_NL, telah ditambahkan. Prioritasnya adalah BROADCAST > MERGE > SHUFFLE_HASH > SHUFFLE_REPLICATE_NL. Jika BROADCAST atau SHUFFLE_HASH ditambahkan ke kedua sisi Gabung, Spark akan memilih sisi mana yang akan dibangun berdasarkan joinType dan ukuran kedua sisi.

-- Join Hints for broadcast join
SELECT /*+ BROADCAST(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;
SELECT /*+ BROADCASTJOIN (t1) */ * FROM t1 left JOIN t2 ON t1.key = t2.key;
SELECT /*+ MAPJOIN(t2) */ * FROM t1 right JOIN t2 ON t1.key = t2.key;
 
-- Join Hints for shuffle sort merge join
SELECT /*+ SHUFFLE_MERGE(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;
SELECT /*+ MERGEJOIN(t2) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;
SELECT /*+ MERGE(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;
 
-- Join Hints for shuffle hash join
SELECT /*+ SHUFFLE_HASH(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;
 
-- Join Hints for shuffle-and-replicate nested loop join
SELECT /*+ SHUFFLE_REPLICATE_NL(t1) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;
 
-- When different join strategy hints are specified on both sides of a join, Spark
-- prioritizes the BROADCAST hint over the MERGE hint over the SHUFFLE_HASH hint
-- over the SHUFFLE_REPLICATE_NL hint.
-- Spark will issue Warning in the following example
-- org.apache.spark.sql.catalyst.analysis.HintErrorLogger: Hint (strategy=merge)
-- is overridden by another hint and will not take effect.
SELECT /*+ BROADCAST(t1), MERGE(t1, t2) */ * FROM t1 INNER JOIN t2 ON t1.key = t2.key;

Hubungan penggunaan dalam petunjuk percikan: https://blog.51cto.com/u_15435003/5296344