Senior Data Engineer (Python, Spark, AWS,SQL)
Location: McLean, VA
Seeking a Senior Full Stack Engineer who enjoys data and building data storage platforms from ground up using microservices architecture. The ideal candidate has a passion for data analysis, technology and helping people leverage the technology to transform their business processes and analytics. As a Data Engineer, you will be part of a team responsible for supporting a wide range of internal customers. You will draw on all the skills in your toolkit to analyze, design, and develop data storage and data analytic solutions using data lake patterns, that help our customers run more effective operations and make better business decisions.
Your Work Falls Into Two Primary Categories:
Strategy Development and Implementation
• Develop data governance related services using SpringBoot and Angular
• Develop data filtering, transformational and loading requirements
• Define and execute ETLs using Apache Spark among other Data technologies
• Determine appropriate translations and validations between source data and target databases
• Implement business logic to cleanse & transform data
• Design and implement appropriate error handling procedures
• Develop project, documentation and storage standards in conjunction with data architects
• Monitor performance, troubleshoot and tune ETL processes as appropriate using tools like in the AWS ecosystem.
• Execution of end to end implementation of underlying data ingestion workflow.
Operations and Technology
• At least 4+ years of experience developing RESTFUL APIs with SpringBoot
• At least 5 years of experience developing in Python and Spark (Java is preferred)
• At least 3 years experience in one of the following Cloud technologies: Amazon Web Services (AWS), Google
• Experience working with different Databases and understanding of data concepts (including data warehousing, data lake patterns, structured and unstructured data)
• Implementation and tuning experience specifically using Amazon Elastic Map Reduce (EMR).
• Implementing AWS services in a variety of distributed computing, enterprise environments.
• Experience writing automated unit, integration, regression, performance and acceptance tests
• Good understanding of the current trends in Data Engineering domain
• Solid understanding of software design principles
Key to success in this role
• Strong consultation and communication skills
• Ability to work with and collaborate across the team and where silos exist
• Deep curiosity to learn about new trends and how to do things better
• Ability to use data to help inform strategy and direction
Top Personal Competencies to possess
• Seek and Embrace Change – Continuously improve work processes rather than accepting the status quo
• Growth and Development – Know or learn what is needed to deliver results and successfully compete