• views: 106
• date: 29 September 2024
• posted by: 0nelove

What you'll learn
bigquery
gcs
hadoop
cloud functions
Requirements
Basic Hadoop knowledge
Description
Objective of the course , is to migrate data from On-prem Hadoop(Consider hadoop installed in Windows is considered as on-prem) to GCS(Google Cloud storage) and Google cloud storage to BigQuery. To learn about this course , basic knowledge in Hadoop commands is mandatory. Things you will learn from this course - Hadoop installation in Windows 11 - Load data from local file system to Hadoop - Load file from hdfs to gcs by installing gcs connector - Once data is loaded in bucket, the bucket name and file name is captured in bigquery table by creating a trigger using cloud function gen2. The input to apache beam is from latest file name bigquery table and load the contents of the file from the bucket to the actual bigquery table. We also tried to create hive external table, which is pointing out GCS bucket and file. Due to errors , we can't able to demo the approach. By creating that , the hive external table is loaded which indirectly loads data in GCS. The apache beam code which loads data from gcs to bigquery will run by direct runner , we faced some errors while running through dataflow runner. The datatype conversion is not handled in migration , considering all the columns as string.
Overview
Section 1: Hadoop to gcp migration - part1
Lecture 1 Hadoop to GCP Migration Using DistCp
Lecture 2 Capture GCS Events in BIgquery
Lecture 3 File from Bucket to BigQuery Table
Lecture 4 Overall flow
Lecture 5 Google cloud Data engineer certification exam experience
Lecture 6 Thank you
GCP Aspirants
Screenshots



   
   
   

Forex Trading

Crypto Trading. 20 USD Bonus after KYC

We need your support!
Make a donation to help us stay online
        
Bitcoin (BTC)
bc1q08g9d22cxkawsjlf8etuek2pc9n2a3hs4cdrld
	
Bitcoin Cash (BCH)
qqvwexzhvgauxq2apgc4j0ewvcak6hh6lsnzmvtkem

Ethereum (ETH)
0xb55513D2c91A6e3c497621644ec99e206CDaf239

Litecoin (LTC)
ltc1qt6g2trfv9tjs4qj68sqc4uf0ukvc9jpnsyt59u

USDT (ERC20)
0xb55513D2c91A6e3c497621644ec99e206CDaf239

USDT (TRC20)
TYdPNrz7v1P9riWBWZ317oBgJueheGjATm

0 1 2 3 4 5

| Print version |

	Information
		Users of GUESTS are not allowed to comment this publication.