Deskripsi Pekerjaan:

As our Astronaut, your missions are:

• Develop a data pipeline for web scraping from various sources
• Develop a data pipeline to transform data into a suitable form for analysis
• Develop automation and instrumentation for deployment, monitoring, alerting, and debugging to ensure reliable pipelines
• Maintain and optimize the cost and performance of our data platform infrastructure (VM, data warehouse, data processor, orchestrator, CDC, message broker, etc)
• Monitor and support day-to-day operations of the data pipeline
• Ensure high data quality, integrity, and reliability

Kualifikasi:

To launch successfully, you need to have:

• 2+ years of data engineering or programming experience, preferably with experience in web scraping
• Strong programming skills, with proficiency in Python or other relevant programming languages
• Have experience with web scraping techniques and tools, such as Selenium, BeautifulSoup, and other modules in Python
• Experienced with cloud infrastructure systems, such as Google Cloud Platform (GCP) Cloud Functions, Cloud Run, Compute Engine, GKE, VPC, Cloud Composer (Apache Airflow), Pub/Sub, BigQuery, etc.
• Proficient in Structured Query Language (SQL)