IT, Data Group, Data Management and Operations Team, Senior Data Engineer, GG11

日付: 2023/11/21

場所: Tokyo, JP

会社: MetLife

Job Description:


Senior Data Engineer is responsible for building, optimizing and maintaining data pipelines that ingest and transform large dataset on cloud.



  • Analyze business requirements, create proper data solutions that meets functional and non-functional requirements
  • Define the data architecture, structure, storage, security model of data in Hadoop and Azure data platform.
  • Design and implement highly scalable data pipelines that ingest data from various SORs, clean, consolidate and transform the data into standard data models, and load them for consumptions and analytics.
  • Create and maintain reusable frameworks for data ingestion, validation, normalization, and transformation.
  • Automate Unit testing and release with CI/CD pipeline
  • Create scripts for operations, releases, data migrations, data validations
  • Maintain data lineage information
  • Provide technical support for releases, issues, and incidents




  • Minimum 3 years of working experience as data engineer or similar role



  • Good understanding of Software Development Lifecycle
  • Solid knowledge and experience of Hadoop ecosystem and Azure data platform: ADLS, Synapse, Azure Cosmos DB, Azure SQL Server
  • Strong programming skills in Scala and Spark
  • Strong capabilities in designing and understanding database schema and data models
  • Good knowledge of Oracle database
  • Proficient in SQL, shell scripting, git, unit testing, CI/CD tools
  • Understanding of design principles, coding standards and best practices
  • Good problem-solving skills and analytics skills
  • Excellent communication, presentation, and interpersonal skills



  • Understanding of REST API, ETL tools, and Salesforce is preferred
  • Understanding of streaming data processing and change data capture tools (CDC) is preferred
  • Experience with BI tools is preferred


  • Japanese: optional but preferred
  • English: intermediate



  • Detailed and precise, able to drive alone the design and development of complex system.
  • Able to articulate technical aspects in an easy-to-understand way for non-developers
  • Strong performer with focus on delivery, fast learning and able to apply his knowledge in various domains.
  • Willing to learn new technologies and pick up challenging work
  • Team player