Data Engineer

Data Engineer Company: Niche Skills Limited Location: Hybrid Employment Type: Full-Time Salary / Day Rate: Competitive, based on experience Right to Work: Applicants must have the right to work in the UK or meet the applicable sponsorship requirements. About Niche Skills Niche Skills Limited is a UK-based technology consultancy delivering modern data, cloud, analytics, automation, and Artificial Intelligence solutions. We help organisations improve how they collect, manage, analyse, and use data. Our consultants work closely with clients to design scalable data platforms, build automated workflows, create business intelligence solutions, and implement AI-powered applications that support better decision-making and digital transformation. We are looking for an experienced and innovative IT Consultant – Data Engineering, Data Analytics \& AI to join our growing team. About the Role You will work across the complete data lifecycle, from data ingestion and engineering through to analytics, machine learning, Generative AI, and intelligent automation. The role combines hands-on technical delivery with consultancy and stakeholder engagement. You will work with clients to understand their business challenges, translate requirements into practical technology solutions, and deliver secure, scalable, and governed data and AI platforms. You may contribute to projects involving Microsoft Fabric, Azure Data Factory, Databricks, Power BI, Azure OpenAI, machine learning, Retrieval-Augmented Generation, and AI-powered business automation. Key ResponsibilitiesConsulting and Solution Design * Engage with clients and business stakeholders to understand their objectives, processes, and data requirements. * Translate business needs into scalable data, analytics, and AI solutions. * Produce solution designs, architecture diagrams, technical documentation, and implementation roadmaps. * Recommend best practices for data architecture, governance, security, cloud adoption, and responsible AI. * Identify opportunities to improve operations through automation, analytics, and Artificial Intelligence. * Provide technical leadership throughout project delivery. Data Engineering * Design, develop, and maintain scalable ETL and ELT data pipelines. * Integrate data from databases, APIs, files, cloud applications, and external systems. * Build and manage data warehouses, data lakes, lakehouses, and cloud-based data platforms. * Develop data models for reporting, analytics, machine learning, and AI workloads. * Implement data validation, quality, governance, security, and monitoring controls. * Troubleshoot and optimise data pipelines, databases, and processing workloads. * Automate data ingestion and transformation processes. Data Analytics and Business Intelligence * Analyse complex datasets to identify business trends, risks, opportunities, and performance issues. * Develop interactive Power BI dashboards, executive reports, KPIs, and analytical models. * Work with stakeholders to define reporting requirements and business metrics. * Perform descriptive, diagnostic, predictive, and prescriptive analytics. * Present findings clearly to technical and non-technical audiences. * Deliver actionable insights that support strategic and operational decision-making. Artificial Intelligence and Automation * Design and implement machine-learning and AI solutions for real business challenges. * Develop predictive models, forecasting solutions, recommendation engines, and intelligent workflows. * Build Generative AI solutions using Large Language Models. * Implement AI assistants, chatbots, document intelligence, semantic search, and Retrieval-Augmented Generation solutions. * Integrate Azure OpenAI and other AI services into business applications and data platforms. * Evaluate model performance and improve accuracy, reliability, and usability. * Apply responsible AI principles, including security, transparency, privacy, human oversight, and ethical use. Collaboration and Delivery * Work closely with developers, architects, project managers, business analysts, and client teams. * Participate in Agile and Scrum ceremonies. * Maintain clear technical and operational documentation. * Support testing, deployment, production readiness, and knowledge transfer. * Mentor junior consultants and share technical knowledge. * Stay current with developments in cloud data platforms, analytics, machine learning, and Generative AI. Required Experience * At least five years of professional experience in data engineering, data analytics, Artificial Intelligence, cloud technology, or IT consulting. * Proven experience delivering end-to-end data and analytics solutions. * Strong understanding of enterprise data architecture and modern cloud-based data platforms. * Experience gathering requirements and working directly with business stakeholders. * Strong analytical, communication, presentation, and problem-solving skills. * Ability to translate business requirements into practical technical solutions. * Experience producing technical documentation and solution designs. Technical Skills Candidates should have strong experience in several of the following areas: Programming and Data Processing * Python * SQL * PySpark * PowerShell or Bash Microsoft Data Platform * Microsoft Fabric * Azure Data Factory * Fabric Data Factory * Azure Synapse Analytics * Azure Databricks * Apache Spark * OneLake * Data pipelines and data integration Databases and Data Architecture * Microsoft SQL Server * Azure SQL Database * PostgreSQL * MySQL * Oracle Database * Data warehouse design * Data lake and lakehouse architecture * Dimensional modelling * Star and snowflake schemas Analytics and Reporting * Power BI * DAX * Power Query * Advanced Microsoft Excel * Tableau experience is desirable Artificial Intelligence * Machine learning * Predictive analytics * Azure AI Services * Azure OpenAI Service * Microsoft Copilot Studio * Prompt engineering * Large Language Models * Retrieval-Augmented Generation * Vector databases * AI agents * Intelligent automation * Document intelligence DevOps and Delivery * Git * GitHub * Azure DevOps * CI/CD pipelines * Infrastructure and deployment automation Experience with AWS or Google Cloud Platform would also be beneficial. What We Are Looking For We are looking for someone who: * Combines strong technical ability with a consulting mindset. * Is confident communicating with clients and senior stakeholders. * Can independently investigate complex business and technical problems. * Understands how to balance innovation with security, governance, and maintainability. * Can manage multiple priorities and deliver high-quality work. * Is collaborative, self-motivated, and results-oriented. * Is committed to continuous learning and professional development. Example Project The successful candidate may contribute to the delivery of an Intelligent Customer Operations Data Platform. The project could involve: * Integrating customer, sales, service, operational, and document data. * Building a Microsoft Fabric lakehouse using a medallion data architecture. * Developing automated data pipelines and data-quality controls. * Creating executive and operational Power BI dashboards. * Building predictive models for customer churn, service demand, and operational risk. * Developing an Azure OpenAI-powered assistant using Retrieval-Augmented Generation. * Implementing security, governance, monitoring, and responsible AI controls. * Deploying solutions through automated CI/CD pipelines. Key Deliverables * Secure and scalable data pipelines. * Modern data warehouse, lakehouse, and cloud data solutions. * Governed and reusable enterprise data models. * Interactive Power BI reports and dashboards. * Predictive machine-learning models. * Generative AI and intelligent automation solutions. * Technical documentation and architecture designs. * Actionable insights that support business decision-making. Why Join Niche Skills? * Work on modern data, analytics, cloud, and Artificial Intelligence projects. * Use technologies including Microsoft Fabric, Azure, Power BI, Azure OpenAI, and Databricks. * Work directly with clients and influence solution design. * Gain exposure to projects across the complete data and AI lifecycle. * Join a collaborative and growing technology consultancy. * Access opportunities for professional development and technical leadership. * Benefit from flexible hybrid, remote, and client-site working arrangements. How to Apply Please submit your CV along with a brief summary of your experience in data engineering, analytics, cloud platforms, and Artificial Intelligence. Where possible, include examples of relevant projects, technologies used, your responsibilities, and the business outcomes delivered. Niche Skills Limited is committed to creating an inclusive working environment. Applications are welcomed from suitably qualified candidates from all backgrounds. Pay: £39,000.00-£40,000.00 per year Benefits: * UK visa sponsorship Work Location: Hybrid remote in Watford (Hertfordshire, East of England Region)